| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Replace kpartx with qemu-nbd in extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/828617 | 01:01 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: rhel: work around RHEL-9 BLS issues https://review.opendev.org/c/openstack/diskimage-builder/+/829620 | 01:01 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Move grub-install to the end, and skip for partition images https://review.opendev.org/c/openstack/diskimage-builder/+/826976 | 01:01 |
| kevinz_ | frickler: Yes now it is refreshed. | 01:43 |
| fungi | thanks kevinz_! | 01:44 |
| kevinz_ | Yw | 01:44 |
| *** ysandeep|out is now known as ysandeep | 01:59 | |
| *** pojadhav|out is now known as pojadhav | 03:50 | |
| *** ysandeep is now known as ysandeep|afk | 03:54 | |
| *** ysandeep|afk is now known as ysandeep | 05:07 | |
| dpawlik | clarkb, fungi: hey. I find out what was going on with log workflow - services were out of memory and the gearman worker (what is interesting) was sending empty messages to logstash service. Today I check the traffic with tcpdump -X parameter. Earlier without -X all was "fine". | 07:50 |
| dpawlik | I added swap file and I will monitoring that. If all will be fine we can ask others to check the service | 07:50 |
| dpawlik | or maybe rr has done something | 07:54 |
| *** amoralej|off is now known as amoralej | 08:16 | |
| *** jpena|off is now known as jpena | 08:36 | |
| *** ysandeep is now known as ysandeep|lunch | 08:41 | |
| opendevreview | Joan Gilabert proposed zuul/zuul-jobs master: Add patch to list of build dependencies https://review.opendev.org/c/zuul/zuul-jobs/+/829063 | 09:16 |
| *** pojadhav is now known as pojadhav|lunch | 09:27 | |
| *** ysandeep|lunch is now known as ysandeep | 09:36 | |
| *** pojadhav|lunch is now known as pojadhav | 10:13 | |
| *** dviroel|out is now known as dviroel | 11:00 | |
| *** rlandy|out is now known as rlandy|ruck | 11:13 | |
| *** ysandeep is now known as ysandeep|break | 12:22 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 12:43 | |
| *** pojadhav is now known as pojadhav|afk | 12:53 | |
| NeilHanlon | morning all :) I think https://review.opendev.org/c/openstack/project-config/+/828435 is ready to proceed if anyone has some spare cycles | 13:10 |
| *** amoralej is now known as amoralej|lunch | 13:21 | |
| dpawlik | clarkb, fungi: seems that gearman worker is a bottleneck. We will work on improving the workflow | 13:30 |
| *** sboyron_ is now known as sboyron | 13:33 | |
| NeilHanlon | hm, this appears to be py3 code in a py2 env: https://zuul.opendev.org/t/openstack/build/cf6a7c769b614efea6a20a6dea58da68 | 13:42 |
| frickler | infra-root: looks like project-config-grafyaml is still using py2.7, not sure if the failure is actually related, but we should include that on our list of things needing updates https://zuul.opendev.org/t/openstack/build/cf6a7c769b614efea6a20a6dea58da68 | 13:43 |
| NeilHanlon | oh hey, jinx :P | 13:43 |
| frickler | or what NeilHanlon says ;) | 13:43 |
| NeilHanlon | https://opendev.org/openstack/project-config/src/branch/master/tox.ini#L87 | 13:45 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: remove basepython override for grafyaml https://review.opendev.org/c/openstack/project-config/+/829712 | 13:47 |
| frickler | https://pypi.org/project/python-slugify/ still builds py2.py3 wheels, so pip tries to install it into py2.7. and they had a new release yesterday | 13:47 |
| frickler | NeilHanlon: o.k., that line was from 2015, but seems it might really be that simple | 13:50 |
| NeilHanlon | that's sorta what I'm thinking, but more context from infra-root is probably needed | 13:50 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration https://review.opendev.org/c/openstack/project-config/+/828435 | 13:51 |
| NeilHanlon | or pin python-slugify in grafyaml to <5.0.0 | 13:54 |
| frickler | NeilHanlon: running locally works just fine for me with py3.9, so I'd prefer that path | 13:54 |
| opendevreview | Neil Hanlon proposed opendev/grafyaml master: Pin python-slugify to <6.0.0 https://review.opendev.org/c/opendev/grafyaml/+/829714 | 13:58 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration https://review.opendev.org/c/openstack/project-config/+/828435 | 14:00 |
| frickler | NeilHanlon: maybe I used the wrong wording, what I prefer is 829712 | 14:04 |
| frickler | sorry for the confusion | 14:04 |
| NeilHanlon | ah, no problem--my misunderstanding. I should have clarified with you | 14:10 |
| NeilHanlon | restored 829712 | 14:10 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: remove basepython override for grafyaml https://review.opendev.org/c/openstack/project-config/+/829712 | 14:11 |
| NeilHanlon | and fixed the commit message a bit | 14:11 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration https://review.opendev.org/c/openstack/project-config/+/828435 | 14:16 |
| *** amoralej|lunch is now known as amoralej | 14:19 | |
| frickler | NeilHanlon: btw., you could also have rebased 828435 on top of 829712, that way you wouldn't have lost the existing votes | 14:30 |
| NeilHanlon | ah, I didn't realize that | 14:33 |
| opendevreview | Merged openstack/project-config master: remove basepython override for grafyaml https://review.opendev.org/c/openstack/project-config/+/829712 | 14:37 |
| frickler | NeilHanlon: for changes within the same repo that is my preferred way of doing things, since also gerrit then shows a list of all patches in a stack and thus makes navigation easier | 14:40 |
| opendevreview | Merged openstack/project-config master: Add rockylinux-8 to nodepool configuration https://review.opendev.org/c/openstack/project-config/+/828435 | 14:41 |
| NeilHanlon | gotcha, makes a lot of sense. I'm still slowly learning gerrit more | 14:41 |
| NeilHanlon | TY for the help | 14:41 |
| *** ysandeep|break is now known as ysandeep | 14:42 | |
| NeilHanlon | is there a way to monitor the nodepool rollout to check for any issues? | 14:45 |
| fungi | NeilHanlon: image build logs are published from https://nb01.opendev.org/ and https://nb02.opendev.org/ | 14:51 |
| NeilHanlon | fungi: ack ty! | 14:56 |
| frickler | there's a failure already in https://nb01.opendev.org/rockylinux-8-0000000001.log which needs to be looked at I think | 15:33 |
| NeilHanlon | taking a look, maybe something different about how it's built on nodepools than in CI for DIB? | 16:04 |
| *** ysandeep is now known as ysandeep|out | 16:19 | |
| clarkb | NeilHanlon: the CI builds tend to be a lot simpler due to time constraints | 16:20 |
| fungi | package coreutils-8.30-12.el8.x86_64 conflicts with coreutils-single provided by coreutils-single-8.30-12.el8.x86_64 | 16:22 |
| fungi | that's... neat | 16:22 |
| fungi | yeah, so that's probably why it didn't show up in ci jobs for dib... we add coreutils in our infra-package-needs element | 16:23 |
| fungi | https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/infra-package-needs/package-installs.yaml#L2 | 16:24 |
| fungi | i guess rocky bootstrapping ends up with coreutils-single installed and then we try to install coreutils and they're not allowed to both be installed in the same system but dnf prefers to error rather than resolve it? | 16:25 |
| fungi | i'm not positive what the solution should be, but maybe map coreutils to coreutils-single in here? https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/infra-package-needs/pkg-map#L16-L22 | 16:27 |
| fungi | NeilHanlon: ^ do you happen to know the difference between those packages and whether they're interchangeable? | 16:27 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove mirror ports 4444 and 8081 from the firewall https://review.opendev.org/c/opendev/system-config/+/829744 | 16:33 |
| opendevreview | Merged opendev/system-config master: Update Interop doc https://review.opendev.org/c/opendev/system-config/+/816982 | 16:48 |
| *** marios is now known as marios|out | 16:51 | |
| NeilHanlon | fungi: as I understand it, coreutils-single is a slimmed down version of coreutils for containers, basically. Probably the right think to do would be for me to change the rockylinux-container element to swap coreutils in for -single, since these are not containers | 16:52 |
| NeilHanlon | s/think/thing | 16:52 |
| fungi | ahh | 16:52 |
| fungi | got it, so this is probably an upshot of our containerfile method | 16:53 |
| NeilHanlon | yeah, a bit. i'll do a quick compare between the kickstarts for container vs cloud and see if there's something else we might want, too | 16:54 |
| clarkb | do we need to pause the builds in the meantime? | 16:54 |
| clarkb | iirc that is easy with a command now | 16:54 |
| fungi | i can | 16:55 |
| NeilHanlon | Thank you | 16:55 |
| fungi | i've done a `nodepool image-pause rockylinux-8` | 16:55 |
| clarkb | thanks | 16:56 |
| fungi | need to remember to image-unpause it when we're ready to resume trying | 16:56 |
| NeilHanlon | coreutils-single seems to be the only problematic bit | 16:58 |
| opendevreview | Neil Hanlon proposed openstack/diskimage-builder master: Swap coreutils-single for coreutils in Rocky Linux container builds https://review.opendev.org/c/openstack/diskimage-builder/+/829747 | 17:00 |
| clarkb | NeilHanlon: fungi: I wont approve that yet since the rocky builds appear to be non voting. But once we get some signal back we can likely land that | 17:04 |
| fungi | thanks | 17:04 |
| fungi | once it lands we'll need a new dib release request, then after that merges we need to increase the minimum dib in nodepool's requirements, then after that merges and builds nodepool's container images we should watch them get pulled and restarted on the builders, after which we can unpause? | 17:06 |
| clarkb | that sounds right | 17:06 |
| *** artom__ is now known as artom | 17:11 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 17:12 | |
| *** eandersson8 is now known as eandersson | 17:19 | |
| NeilHanlon | clarkb: ack, thanks! | 17:27 |
| fungi | NeilHanlon: another tip, for a quick change like that, making sure your branch is up to date with the current state of master first means that, assuming no revisions are needed and nothing else merges in the interim, we can know in advance what the commit id might be (don't rebase that one though, just a suggestion for the future) | 17:32 |
| fungi | if it had been based on current master, i'd have gone ahead and prepped a release request referencing that commit | 17:32 |
| NeilHanlon | oh, I thought I had rebased it to the tip | 17:32 |
| clarkb | fungi: releases are not managed that way | 17:33 |
| clarkb | fungi: this is intentioanl because it is a chicken and egg to have dib released by the release tooling | 17:33 |
| fungi | oh, is dib independent release? okay, that's easier then | 17:33 |
| *** jpena is now known as jpena|off | 17:34 | |
| fungi | looks like it's had a doc update, some bootloader cleanup, and netifs names fallback changes merged since the last tag, so all minor fixes from the looks of it... should be able to release as 3.18.1 once that change merges | 17:36 |
| clarkb | it is probably worth double checking with ianw before making a release just in case the bootloader stuff is halfway done or there is some additional needed update | 17:37 |
| fungi | yep, he's the one who's pushed all the recent tags, so i figure we can bring it up with him once he's awake/around | 17:38 |
| opendevreview | Merged opendev/system-config master: Remove mirror ports 4444 and 8081 from the firewall https://review.opendev.org/c/opendev/system-config/+/829744 | 17:58 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/829615 and https://review.opendev.org/c/openstack/project-config/+/829121 are another couple of easy reviews if you have a moment | 18:07 |
| *** sshnaidm is now known as sshnaidm|afk | 18:17 | |
| fungi | sure thing | 18:17 |
| *** amoralej is now known as amoralej|off | 18:28 | |
| opendevreview | Merged openstack/project-config master: Set noop jobs on opendev repos for retirement https://review.opendev.org/c/openstack/project-config/+/829121 | 18:35 |
| clarkb | apparently when browser versions enter 3 digit territory (they are near 100 right now at 98ish) a bunch of things are expected to break | 18:51 |
| clarkb | I assume this is somethign we can test if we are worried about it by setting custom user agent values | 18:52 |
| clarkb | btu also I run firefox beta and maybe that is good enough to catch things early | 18:52 |
| *** kopecmartin_ is now known as kopecmartin | 19:11 | |
| corvus | i plan on restarting zuul-web after lunch | 19:22 |
| opendevreview | Clark Boylan proposed opendev/puppet-apparmor master: Retire this repo https://review.opendev.org/c/opendev/puppet-apparmor/+/829759 | 19:23 |
| clarkb | corvus: thanks for the heads up | 19:23 |
| clarkb | I'm going to start working on writing a bunch of ^ those changes | 19:24 |
| opendevreview | Merged opendev/system-config master: Reload haproxy when its config updates https://review.opendev.org/c/opendev/system-config/+/829615 | 19:25 |
| opendevreview | Clark Boylan proposed opendev/puppet-askbot master: Retire this repo https://review.opendev.org/c/opendev/puppet-askbot/+/829760 | 19:26 |
| opendevreview | Clark Boylan proposed opendev/puppet-asterisk master: Retire this repo https://review.opendev.org/c/opendev/puppet-asterisk/+/829761 | 19:29 |
| opendevreview | Clark Boylan proposed opendev/puppet-bandersnatch master: Retire this repo https://review.opendev.org/c/opendev/puppet-bandersnatch/+/829762 | 19:30 |
| opendevreview | Clark Boylan proposed opendev/puppet-bugdaystats master: Retire this repo https://review.opendev.org/c/opendev/puppet-bugdaystats/+/829763 | 19:32 |
| opendevreview | Clark Boylan proposed opendev/puppet-ciwatch master: Retire this repo https://review.opendev.org/c/opendev/puppet-ciwatch/+/829764 | 19:34 |
| opendevreview | Clark Boylan proposed opendev/puppet-diskimage_builder master: Retire this repo https://review.opendev.org/c/opendev/puppet-diskimage_builder/+/829766 | 19:36 |
| opendevreview | Clark Boylan proposed opendev/puppet-germqtt master: Retire this repo https://review.opendev.org/c/opendev/puppet-germqtt/+/829767 | 19:38 |
| ianw | i think dib is releasable, but it seems 829747 failed | 19:38 |
| fungi | yeah, looks like NeilHanlon wanted to discuss some possible solutions | 19:38 |
| opendevreview | Clark Boylan proposed opendev/puppet-grafyaml master: Retire this repo https://review.opendev.org/c/opendev/puppet-grafyaml/+/829768 | 19:40 |
| opendevreview | Clark Boylan proposed opendev/puppet-graphite master: Retire this repo https://review.opendev.org/c/opendev/puppet-graphite/+/829769 | 19:42 |
| opendevreview | Clark Boylan proposed opendev/puppet-haveged master: Retire this repo https://review.opendev.org/c/opendev/puppet-haveged/+/829790 | 19:43 |
| opendevreview | Clark Boylan proposed opendev/puppet-hound master: Retire this repo https://review.opendev.org/c/opendev/puppet-hound/+/829791 | 19:44 |
| opendevreview | Clark Boylan proposed opendev/puppet-infra-cookiecutter master: Retire this repo https://review.opendev.org/c/opendev/puppet-infra-cookiecutter/+/829793 | 19:46 |
| opendevreview | Clark Boylan proposed opendev/puppet-jenkins master: Retire this repo https://review.opendev.org/c/opendev/puppet-jenkins/+/829794 | 19:47 |
| opendevreview | Clark Boylan proposed opendev/puppet-kerberos master: Retire this repo https://review.opendev.org/c/opendev/puppet-kerberos/+/829795 | 19:49 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Swap coreutils-single for coreutils in Rocky Linux container builds https://review.opendev.org/c/openstack/diskimage-builder/+/829747 | 19:50 |
| opendevreview | Clark Boylan proposed opendev/puppet-lodgeit master: Retire this repo https://review.opendev.org/c/opendev/puppet-lodgeit/+/829796 | 19:51 |
| ianw | NeilHanlon: ^^ it just looks to me like a missing "-y", but i haven't thought about the package conflict yet. where were you seeing that other error? | 19:51 |
| opendevreview | Clark Boylan proposed opendev/puppet-lpmqtt master: Retire this repo https://review.opendev.org/c/opendev/puppet-lpmqtt/+/829798 | 19:52 |
| opendevreview | Clark Boylan proposed opendev/puppet-mailman master: Retire this repo https://review.opendev.org/c/opendev/puppet-mailman/+/829799 | 19:53 |
| opendevreview | Clark Boylan proposed opendev/puppet-mediawiki master: Retire this repo https://review.opendev.org/c/opendev/puppet-mediawiki/+/829800 | 19:55 |
| opendevreview | Clark Boylan proposed opendev/puppet-meetbot master: Retire this repo https://review.opendev.org/c/opendev/puppet-meetbot/+/829801 | 19:57 |
| opendevreview | Clark Boylan proposed opendev/puppet-mosquitto master: Retire this repo https://review.opendev.org/c/opendev/puppet-mosquitto/+/829802 | 19:58 |
| opendevreview | Clark Boylan proposed opendev/puppet-mqtt_statsd master: Retire this repo https://review.opendev.org/c/opendev/puppet-mqtt_statsd/+/829803 | 20:00 |
| clarkb | Sorry about the spam. Thats about half of them so far. I need lunch now though so taking a break | 20:00 |
| clarkb | ianw: I think the error was in the nodepool image build logs that we build on either nb01 or nb02 (or maybe both) | 20:00 |
| fungi | cleanup is never spam | 20:02 |
| NeilHanlon | ianw: good catch, thanks! let me find the log | 20:42 |
| opendevreview | Clark Boylan proposed opendev/puppet-nodepool master: Retire this repo https://review.opendev.org/c/opendev/puppet-nodepool/+/829806 | 20:42 |
| opendevreview | Clark Boylan proposed opendev/puppet-openafs master: Retire this repo https://review.opendev.org/c/opendev/puppet-openafs/+/829807 | 20:44 |
| NeilHanlon | ianw: https://paste.opendev.org/show/bURxRyhjLJP9V2qF7xNV/ | 20:44 |
| ianw | NeilHanlon: oh, sorry, i meant the grub error from the other change | 20:45 |
| NeilHanlon | ah | 20:45 |
| opendevreview | Neil Hanlon proposed openstack/diskimage-builder master: Swap coreutils-single for coreutils in Rocky Linux container builds https://review.opendev.org/c/openstack/diskimage-builder/+/829747 | 20:46 |
| opendevreview | Clark Boylan proposed opendev/puppet-openstackci master: Retire this repo https://review.opendev.org/c/opendev/puppet-openstackci/+/829808 | 20:46 |
| opendevreview | Clark Boylan proposed opendev/puppet-pgsql_backup master: Retire this repo https://review.opendev.org/c/opendev/puppet-pgsql_backup/+/829809 | 20:48 |
| NeilHanlon | lol.. forgot to refresh before submitting that update. was already fixed by frickler! | 20:48 |
| opendevreview | Clark Boylan proposed opendev/puppet-planet master: Retire this repo https://review.opendev.org/c/opendev/puppet-planet/+/829810 | 20:49 |
| opendevreview | Clark Boylan proposed opendev/puppet-ptgbot master: Retire this repo https://review.opendev.org/c/opendev/puppet-ptgbot/+/829811 | 20:50 |
| opendevreview | Clark Boylan proposed opendev/puppet-puppet master: Retire this repo https://review.opendev.org/c/opendev/puppet-puppet/+/829812 | 20:51 |
| opendevreview | Clark Boylan proposed opendev/puppet-refstack master: Retire this repo https://review.opendev.org/c/opendev/puppet-refstack/+/829814 | 20:53 |
| opendevreview | Clark Boylan proposed opendev/puppet-ssl_cert_check master: Retire this repo https://review.opendev.org/c/opendev/puppet-ssl_cert_check/+/829815 | 20:54 |
| *** dviroel is now known as dviroel|out | 20:56 | |
| opendevreview | Clark Boylan proposed opendev/puppet-statusbot master: Retire this repo https://review.opendev.org/c/opendev/puppet-statusbot/+/829816 | 20:56 |
| opendevreview | Clark Boylan proposed opendev/puppet-sudoers master: Retire this repo https://review.opendev.org/c/opendev/puppet-sudoers/+/829817 | 20:57 |
| opendevreview | Clark Boylan proposed opendev/puppet-translation_checksite master: Retire this repo https://review.opendev.org/c/opendev/puppet-translation_checksite/+/829818 | 20:58 |
| opendevreview | Clark Boylan proposed opendev/puppet-unattended_upgrades master: Retire this repo https://review.opendev.org/c/opendev/puppet-unattended_upgrades/+/829819 | 21:00 |
| opendevreview | Clark Boylan proposed opendev/puppet-unbound master: Retire this repo https://review.opendev.org/c/opendev/puppet-unbound/+/829820 | 21:01 |
| opendevreview | Clark Boylan proposed opendev/puppet-zuul master: Retire this repo https://review.opendev.org/c/opendev/puppet-zuul/+/829821 | 21:02 |
| opendevreview | Clark Boylan proposed opendev/askbot-theme master: Retire this repo https://review.opendev.org/c/opendev/askbot-theme/+/829822 | 21:04 |
| opendevreview | Clark Boylan proposed opendev/germqtt master: Retire this repo https://review.opendev.org/c/opendev/germqtt/+/829823 | 21:07 |
| corvus | restarting zuul-web now | 21:07 |
| opendevreview | Clark Boylan proposed opendev/lpmqtt master: Retire this repo https://review.opendev.org/c/opendev/lpmqtt/+/829824 | 21:08 |
| corvus | #status log restarted zuul-web on commit ba041a3d8ba31355a9057367c6b836589f9fe805 to address log streaming errors | 21:09 |
| opendevstatus | corvus: finished logging | 21:09 |
| opendevreview | Clark Boylan proposed opendev/mqtt_statsd master: Retire this repo https://review.opendev.org/c/opendev/mqtt_statsd/+/829825 | 21:11 |
| clarkb | ok that should be all of the retirement as identified on my etherpad | 21:11 |
| clarkb | Once those land the next step is to mark them retired in projects.yaml and remove their zuul configuration and any gerritbot content | 21:11 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: rhel: work around RHEL-9 BLS issues https://review.opendev.org/c/openstack/diskimage-builder/+/829620 | 21:19 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Replace kpartx with qemu-nbd in extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/828617 | 21:19 |
| clarkb | fungi: did the gerrit rebuild end up happy? | 21:38 |
| fungi | clarkb: it succeeded, yes, and the dnm change tripped the autohold as well... https://173.231.255.108/ | 22:29 |
| fungi | clarkb: though doesn't look like it used your fix. see the gitea link at https://173.231.255.108/c/x/test-project/+/2/1/file-2.txt | 22:30 |
| fungi | still has refs/changes in it | 22:31 |
| fungi | maybe we grabbed the wrong repo state for gerrit? | 22:31 |
| clarkb | fungi: v3.4.3-45-g4045c7c9a6 is the tip of 3.4 so maybe we broke it when we implemented davido's refactoring ... | 23:49 |
| clarkb | bah we did | 23:50 |
| clarkb | arg | 23:50 |
| clarkb | I'm working on a fix | 23:51 |
| *** rlandy|ruck is now known as rlandy|out | 23:53 | |
| fungi | thanks | 23:56 |
| clarkb | fungi: https://gerrit-review.googlesource.com/c/gerrit/+/330599 | 23:59 |
| clarkb | I guess we set a depends on that and triple check it is working then ask davido to rereview | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!