| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Add basic Zuul job https://review.opendev.org/c/opendev/dstat_graph/+/820642 | 00:00 |
|---|---|---|
| clarkb | its up. I can ssh in and the webserver is serving mirrory things | 00:02 |
| clarkb | hopefully the daily letsencrypt run will be happy and renew any certs that need that | 00:02 |
| clarkb | our first gentoo build in a long time is running now. It hasn't failed yet (I looked for logs for the previous ones and they are all months old) | 00:05 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Add .gitreview and basic Zuul job https://review.opendev.org/c/opendev/dstat_graph/+/820609 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Update dependencies https://review.opendev.org/c/opendev/dstat_graph/+/820630 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Trim styles https://review.opendev.org/c/opendev/dstat_graph/+/820631 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Conver to bootstrap 5 https://review.opendev.org/c/opendev/dstat_graph/+/820632 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Trim margin of overview panel https://review.opendev.org/c/opendev/dstat_graph/+/820633 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Fix walking elements of input list https://review.opendev.org/c/opendev/dstat_graph/+/820634 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Use list-group-item for graphs https://review.opendev.org/c/opendev/dstat_graph/+/820635 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Filter out blank lines https://review.opendev.org/c/opendev/dstat_graph/+/820636 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Replace remove .size() with .length https://review.opendev.org/c/opendev/dstat_graph/+/820637 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Fix focus panel chart selection for pcp-dstat https://review.opendev.org/c/opendev/dstat_graph/+/820638 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Update generate_page.sh https://review.opendev.org/c/opendev/dstat_graph/+/820639 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Add a sample.csv and update docs https://review.opendev.org/c/opendev/dstat_graph/+/820640 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/dstat_graph master: Fix refresh for embedded csv https://review.opendev.org/c/opendev/dstat_graph/+/820641 | 00:12 |
| tristanC | clarkb: nice, that's good to hear :-) | 00:16 |
| ianw | clarkb: as the ptl can you quickly add infra core to https://review.opendev.org/admin/repos/opendev/dstat_graph or should i use my admin account? | 00:26 |
| fungi | i expect he has to use his admin account to do that too. either way, it can be done via the ssh api, i think i added an example to the system-config docs | 00:27 |
| ianw | yeah i just wasn't sure what was quicker. i'll pull up my admin details | 00:28 |
| Clark[m] | Yup it's the same process for both of us. I think you can go for it | 00:38 |
| ianw | done, thanks :) | 00:38 |
| opendevreview | Merged opendev/system-config master: Add comments to manage-projects about project-config syncing https://review.opendev.org/c/opendev/system-config/+/815049 | 00:39 |
| opendevreview | Merged opendev/dstat_graph master: Add .gitreview and basic Zuul job https://review.opendev.org/c/opendev/dstat_graph/+/820609 | 00:42 |
| opendevreview | Merged opendev/dstat_graph master: Update dependencies https://review.opendev.org/c/opendev/dstat_graph/+/820630 | 00:54 |
| opendevreview | Merged opendev/dstat_graph master: Trim styles https://review.opendev.org/c/opendev/dstat_graph/+/820631 | 00:54 |
| opendevreview | Merged opendev/dstat_graph master: Conver to bootstrap 5 https://review.opendev.org/c/opendev/dstat_graph/+/820632 | 00:54 |
| opendevreview | Merged opendev/dstat_graph master: Trim margin of overview panel https://review.opendev.org/c/opendev/dstat_graph/+/820633 | 00:54 |
| opendevreview | Merged opendev/dstat_graph master: Fix walking elements of input list https://review.opendev.org/c/opendev/dstat_graph/+/820634 | 00:54 |
| opendevreview | Merged opendev/dstat_graph master: Use list-group-item for graphs https://review.opendev.org/c/opendev/dstat_graph/+/820635 | 00:54 |
| opendevreview | Merged opendev/dstat_graph master: Filter out blank lines https://review.opendev.org/c/opendev/dstat_graph/+/820636 | 00:54 |
| opendevreview | Merged opendev/dstat_graph master: Replace remove .size() with .length https://review.opendev.org/c/opendev/dstat_graph/+/820637 | 01:57 |
| opendevreview | Merged opendev/dstat_graph master: Fix focus panel chart selection for pcp-dstat https://review.opendev.org/c/opendev/dstat_graph/+/820638 | 01:57 |
| opendevreview | Merged opendev/dstat_graph master: Update generate_page.sh https://review.opendev.org/c/opendev/dstat_graph/+/820639 | 02:01 |
| opendevreview | Merged opendev/dstat_graph master: Add a sample.csv and update docs https://review.opendev.org/c/opendev/dstat_graph/+/820640 | 02:01 |
| opendevreview | Merged opendev/dstat_graph master: Fix refresh for embedded csv https://review.opendev.org/c/opendev/dstat_graph/+/820641 | 02:01 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-dstat-graph: clone from opendev https://review.opendev.org/c/zuul/zuul-jobs/+/820645 | 02:06 |
| opendevreview | Ian Wienand proposed openstack/project-config master: cache-devstack: use dstat_graph from opendev https://review.opendev.org/c/openstack/project-config/+/820646 | 02:08 |
| ianw | the gentoo build finished and is uploaded | 03:03 |
| fungi | i'll recheck my changes in that case | 03:04 |
| fungi | rechecked 782339 and 771106 | 03:05 |
| fungi | they'll need a rebase, apparently | 03:07 |
| ianw | i think since https://review.opendev.org/c/zuul/zuul-jobs/+/771105 we also won't try it on gentoo | 03:08 |
| wxy-xiyuan | fungi: https://review.opendev.org/c/openstack/project-config/+/818723 can you take a look at this patch once you're free? Thanks | 03:15 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Set Gentoo profile in configure-mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 03:17 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: Revert "Temporarily stop running Gentoo base role tests" https://review.opendev.org/c/zuul/zuul-jobs/+/771106 | 03:17 |
| ianw | the gentoo images built, but do not seem happy | 04:07 |
| ianw | !!! Your current profile is invalid. If you have just changed your profile | 04:07 |
| opendevmeet | ianw: Error: "!!" is not a valid command. | 04:07 |
| ianw | !!! configuration, you should revert back to the previous configuration. | 04:07 |
| opendevmeet | ianw: Error: "!!" is not a valid command. | 04:07 |
| *** pojadhav|out is now known as pojadhav|rover | 04:14 | |
| opendevreview | Merged openstack/diskimage-builder master: Update keylime-agent and tpm-emulator elements https://review.opendev.org/c/openstack/diskimage-builder/+/810254 | 04:29 |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod: setup system-config on bridge in bootstrap job https://review.opendev.org/c/opendev/system-config/+/820320 | 05:24 |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod-base: parent to infra-prod-setup-keys https://review.opendev.org/c/opendev/system-config/+/820651 | 05:24 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: opendev-infra-prod-base: remove unused job https://review.opendev.org/c/opendev/base-jobs/+/820652 | 05:24 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Update bridge playbook match https://review.opendev.org/c/opendev/system-config/+/820281 | 05:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Rename install-ansible to bootstrap-bridge https://review.opendev.org/c/opendev/system-config/+/820282 | 05:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod: setup system-config on bridge in bootstrap job https://review.opendev.org/c/opendev/system-config/+/820320 | 05:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod-base: parent to infra-prod-setup-keys https://review.opendev.org/c/opendev/system-config/+/820651 | 05:28 |
| *** ysandeep|out is now known as ysandeep | 05:28 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod-base: parent to infra-prod-setup-keys https://review.opendev.org/c/opendev/system-config/+/820651 | 05:44 |
| *** marios is now known as marios|ruck | 06:23 | |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add dnf versionlock support https://review.opendev.org/c/openstack/diskimage-builder/+/811945 | 06:44 |
| *** ysandeep is now known as ysandeep|lunch | 07:39 | |
| *** pojadhav|rover is now known as pojadhav|rover|afk | 09:14 | |
| *** ysandeep|lunch is now known as ysandeep | 09:19 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:20 | |
| frickler | sorry for the german link, but TIL that the current snow in Germany is brought to us by ... Gerrit ;) https://www.wetteronline.de/fotostrecken/2021-12-07-ws | 09:29 |
| ianw | gerrrr-iiittt the snow man, was a jolly happy soul | 09:31 |
| ianw | (i have no idea if frosty the snowman is a thing in Germany :) | 09:33 |
| ianw | somehow he's a thing in Australia, where it would be more like frosty the puddle | 09:33 |
| *** pojadhav|rover|afk is now known as pojadhav|rover | 09:51 | |
| *** redrobot8 is now known as redrobot | 10:20 | |
| *** ysandeep is now known as ysandeep|afk | 11:00 | |
| *** rlandy|out is now known as rlandy|ruck | 11:08 | |
| *** ysandeep|afk is now known as ysandeep | 11:46 | |
| *** bhagyashris_ is now known as bhagyashris | 12:31 | |
| *** pojadhav|rover is now known as pojadhav|rover|brb | 12:44 | |
| *** pojadhav|rover|brb is now known as pojadhav|rover | 13:14 | |
| fungi | perhaps it's a compellingly exotic concept for au | 13:48 |
| noonedeadpunk | feels like zuul either super overflowed or a bit stuck? | 14:03 |
| noonedeadpunk | and most likely because of osa branching :( | 14:03 |
| noonedeadpunk | tons of post jobs waiting for hour or so | 14:05 |
| noonedeadpunk | and new ones are not queued for check | 14:05 |
| ykarel | fungi, clarkb hi can you help in getting latest resource usage by projects | 14:17 |
| ykarel | may be for last 15 days, an output like https://paste.opendev.org/show/jD6kAP9tHk7PZr2nhv8h/ | 14:18 |
| ykarel | seems https://opendev.org/zuul/zuul/src/branch/master/tools/node_usage.py is the script for collecting that | 14:18 |
| ykarel | mainly looking for neutron projects, would like to see how much it differs from past report | 14:19 |
| ykarel | as we are currently trying to improve it further so wanted to have checkpoint so we have something to compare | 14:19 |
| ykarel | or if i need to contact someone else for this please redirect | 14:19 |
| *** odyssey4me is now known as Guest7866 | 14:59 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Copy Exim logs in system-config-run jobs https://review.opendev.org/c/opendev/system-config/+/820899 | 15:16 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Block outbound SMTP connections from test jobs https://review.opendev.org/c/opendev/system-config/+/820900 | 15:16 |
| corvus | i'm going to do a rolling restart of zuul | 15:21 |
| corvus | well, the schedulers and web at least | 15:22 |
| fungi | noonedeadpunk: yeah, i think all the releasing/branching for openstackansible is causing delays. the executors are spending a good chunk of time not accepting new builds while they work through the ones they've already started | 15:25 |
| fungi | it does seem to be burning those counts down though, builds are being performed and completing | 15:25 |
| fungi | corvus: thanks for the heads up | 15:27 |
| corvus | tell me about this branching? | 15:34 |
| *** ysandeep is now known as ysandeep|dinner | 15:34 | |
| fungi | corvus: https://review.opendev.org/c/openstack/releases/+/820644 Create Xena branch for OpenStack-Ansible roles | 15:37 |
| fungi | there's a bunch of jobs in the openstack tenant's post pipeline from that because openstackansible has i don't know how many repos but probably well over a hundred | 15:38 |
| corvus | are they all created now? | 15:38 |
| fungi | i haven't looked, but probably so... it's the jobs triggered by the branch creation events which make up the current pileup | 15:38 |
| corvus | right, but the reason the system seems slow is that it's continuously reconfiguring the openstack tenant because of a bunch of new branches | 15:39 |
| corvus | so i was wondering if they were all created already, or is that still in progress | 15:40 |
| fungi | yeah, looks like the tag-releases job for that change has completed | 15:40 |
| fungi | https://zuul.opendev.org/t/openstack/build/d29249c37338448bb98b7a90b4734e10 | 15:40 |
| fungi | finished almost an hour ago | 15:41 |
| fungi | only took a couple minutes | 15:41 |
| fungi | oh, that was a later change i think | 15:41 |
| clarkb | ykarel: this came up in #openstack-infra a couple of weeks ago. As mentioned then that script doesn't work with modern zuul anymore. In particular the log formats have changed and the logs have moved from a centralized scheduler (though we have decentralized that too) to all of the executors. This means it is possible to collect similar data using that tooling but much more | 15:42 |
| clarkb | difficult. What I did do back then was hack up the script to get a sample from a single executor and I should be able to do that again. But I suggested that openstack/neutron/etc look at using the graphite data to build a grafana dashboard for this information instead | 15:42 |
| fungi | corvus: it was this one, finished something like 5.5 hours ago: https://zuul.opendev.org/t/openstack/build/ab256275292a4c0cbcf9f025d7cca21c | 15:42 |
| fungi | looks like it created 55 branches | 15:43 |
| fungi | we seem to have a strangely high count of available (ready) nodes | 15:46 |
| fungi | according to grafana | 15:46 |
| fungi | yeah, nodepool list confirms 441 nodes in "ready" state | 15:47 |
| corvus | that's due to 2 reasons: the constant reconfigurations and at this moment there is no scheduler running | 15:47 |
| fungi | ahh, okay | 15:47 |
| ykarel | clarkb, Thanks, data from 1/2 executor should be fine to start with | 15:48 |
| ykarel | if possible can you run that custom script again, that will be helpful | 15:49 |
| ykarel | wrt graphite/grafana tha needs to be explored | 15:50 |
| fungi | ykarel: to repeat, "that script doesn't work with modern zuul anymore" | 15:51 |
| clarkb | fungi: I mean I did hack it up | 15:52 |
| ykarel | fungi, yeap i got that, but afaiu clarkb said it's possible with hack to get datafrom an executor | 15:52 |
| clarkb | the script that is committed doesn't work ebacuse it assumes the old scheduler log format | 15:53 |
| clarkb | there is a new executor format and it changes hosts | 15:53 |
| fungi | oh, i get what you meant by "custom script" now | 15:53 |
| clarkb | and ya I can run it on a single executor | 15:53 |
| corvus | okay, there is an issue with the data format in the new zuul; i would normally be tempted to just fix it and roll forward, however, if we also delete all the state data, we will effectively collapse all of the branch creation/reconfig events into one. so i'm inclined to perform a delete-state and full restart in order to address both things. | 15:55 |
| clarkb | corvus: the branch creation stuff from osa? | 15:57 |
| corvus | yep | 15:57 |
| fungi | seems like we'd probably get back to building things faster again with the full restart than waiting for however many reconfigs are pending from all the branch creation events | 15:57 |
| corvus | yeah. i'm not sure why they weren't collapsed already; i'll have to look into that. | 15:57 |
| clarkb | ykarel: https://paste.opendev.org/show/811516/ that is from ze01.opendev.org | 15:58 |
| ykarel | clarkb, Thanks can you run with may be 15 days and on seperate executor too like ze02? | 15:59 |
| ykarel | just to have some more data | 15:59 |
| clarkb | ykarel: no, the executors only log 7 days. and I'd prefer to not need to mantain this on a bunch of hosts. We have the data aggregated in graphite, it just needs to be queried there | 16:00 |
| ykarel | clarkb, Thanks | 16:00 |
| ykarel | clarkb++ | 16:01 |
| ykarel | will check graphite too, but may be will ping here probably next week as i am not much aware about it | 16:01 |
| clarkb | corvus: you're also talking about landing a fix and then doing a full restart on the fix? Or rolling back with the delete state? | 16:02 |
| clarkb | ykarel: let me see if I can find logs from the #openstack-infra discussion | 16:02 |
| corvus | clarkb: sorry, issue was a non-backwards-compat change | 16:02 |
| corvus | so just a delete+restart should address it | 16:02 |
| clarkb | got it | 16:02 |
| clarkb | ykarel: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2021-11-15.log.html | 16:04 |
| clarkb | that includes pointers to the graphite stuff | 16:04 |
| ykarel | clarkb, Thanks noted, will have a look | 16:06 |
| clarkb | infra-root I updated https://etherpad.opendev.org/p/xvu2oKUQVLiHIHsvGRUt slightly for gentoo. Does that email look good to send out now? | 16:09 |
| *** ysandeep|dinner is now known as ysandeep | 16:10 | |
| fungi | clarkb: thanks, lgtm | 16:11 |
| corvus | 1 scheduler up, web and 2nd scheduler on their way | 16:22 |
| *** artom__ is now known as artom | 16:22 | |
| corvus | re-enqueing | 16:29 |
| clarkb | ci image email sent | 16:39 |
| corvus | fungi: can you take a look at https://review.opendev.org/820276 and https://review.opendev.org/820277 ? | 16:48 |
| fungi | sure! | 16:48 |
| corvus | we should probably merge those and test that out soon :) | 16:48 |
| fungi | ooh, thanks i missed these | 16:48 |
| opendevreview | Merged openstack/project-config master: Add REST api auth rules https://review.opendev.org/c/openstack/project-config/+/820277 | 17:02 |
| opendevreview | Lajos Katona proposed openstack/project-config master: Fix Neutron periodic dashboard https://review.opendev.org/c/openstack/project-config/+/820912 | 17:03 |
| *** marios|ruck is now known as marios|out | 17:06 | |
| opendevreview | Merged opendev/system-config master: Add local auth provider to zuul https://review.opendev.org/c/opendev/system-config/+/820276 | 17:54 |
| *** ysandeep is now known as ysandeep|out | 18:23 | |
| corvus | re-enqueue complete. (it really took that long) | 18:29 |
| fungi | wow, i guess it was all the post pipeline refs in the openstack tenant? | 18:30 |
| fungi | and thanks! | 18:30 |
| fungi | the failure on https://zuul.opendev.org/t/openstack/build/d11e7de3c5bc4d179bf1f36d661a9f66 is baffling me. the rules seem to include what it's testing for according to https://zuul.opendev.org/t/openstack/build/d11e7de3c5bc4d179bf1f36d661a9f66/log/focal/rules.v4.txt#19 | 18:47 |
| clarkb | fungi: maybe try an explicit print(rules) to see what is in there? | 18:50 |
| clarkb | fungi: also I think that ruleset needs to add openstack-OUTPUT to the OUTPUT chain? | 18:53 |
| fungi | oh, yep! | 18:54 |
| fungi | thanks | 18:54 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Block outbound SMTP connections from test jobs https://review.opendev.org/c/opendev/system-config/+/820900 | 18:57 |
| clarkb | it may be getting optimized out due to that lack of attachment | 18:57 |
| fungi | yep | 18:59 |
| fungi | worth confirming before i debug further anyway. thanks for spotting it | 18:59 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Block outbound SMTP connections from test jobs https://review.opendev.org/c/opendev/system-config/+/820900 | 19:57 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Copy Exim logs in system-config-run jobs https://review.opendev.org/c/opendev/system-config/+/820899 | 19:57 |
| fungi | i'm trying to reverse-engineer the testinfra iptables validation, but suspect i'm still missing something | 20:01 |
| clarkb | I can take a look after some lunch and a bike ride. But need to get out now before it gets rainy and dark | 20:04 |
| clarkb | though it is wetter than I had hoped already :/ | 20:04 |
| fungi | yeah, there's no rush. enjoy! | 20:13 |
| opendevreview | Merged zuul/zuul-jobs master: Update Fedora 34 to 35 https://review.opendev.org/c/zuul/zuul-jobs/+/818702 | 20:30 |
| ianw | infra-root: low priority but https://review.opendev.org/c/zuul/zuul-jobs/+/820645 and https://review.opendev.org/c/openstack/project-config/+/820646 update the dstat graph generation to use our opendev fork instead of github | 20:32 |
| opendevreview | Merged zuul/zuul-jobs master: ensure-dstat-graph: clone from opendev https://review.opendev.org/c/zuul/zuul-jobs/+/820645 | 20:54 |
| opendevreview | Merged opendev/system-config master: Update bridge playbook match https://review.opendev.org/c/opendev/system-config/+/820281 | 20:57 |
| fungi | i have a feeling i'm going to need to set an autohold on 820900 and check whatever command the testinfra iptables module is running, because the output it quotes is uselessly truncated | 21:04 |
| fungi | but i'll wait to see if anyone else can spot the issue before i go to that length | 21:05 |
| ianw | fungi: to start with probably adding some info to that assert would help, should be able to just do assert rule in rules, "Rule %s not seen in rules" % rule ... ish | 21:12 |
| corvus | i'm going to restart zuul-web | 21:17 |
| * TheJulia drifts in | 21:22 | |
| TheJulia | o/ | 21:22 |
| ianw | sigh, gnome has stopped accepting clicks in my work vm, so i'm going to reboot :/ | 21:25 |
| TheJulia | corvus: any luck? | 21:25 |
| TheJulia | okay, its back | 21:28 |
| corvus | yep, just takes a while to start now; we need to add a second one and LB | 21:29 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add zuul-client config to schedulers https://review.opendev.org/c/opendev/system-config/+/820951 | 21:40 |
| corvus | infra-root: ^ i manually did that on zuul02 for development purposes. you can run something like `zuul-client --tenant zuul autohold-list` there now and see it in action | 21:41 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!