Tuesday, 2022-08-02

*** rlandy is now known as rlandy|out00:19
ianwsigh, a gate failure on that02:23
ianwerror: Status code: 503 for https://mirrors.centos.org/metalin ... blah -- upstream failure02:24
wanght26test03:21
*** NeilHanlon_ is now known as NeilHanlon03:57
*** NeilHanlon is now known as neil03:59
*** neil is now known as NeilHanlon03:59
*** diablo_rojo is now known as Guest686104:32
*** ysandeep|out is now known as ysandeep05:18
opendevreviewAnnu Mishra proposed openstack/diskimage-builder master: Add subscription-manager repo names for RHEL-9  https://review.opendev.org/c/openstack/diskimage-builder/+/85173406:28
opendevreviewMerged openstack/diskimage-builder master: Disable the opensuse functest  https://review.opendev.org/c/openstack/diskimage-builder/+/85175906:46
*** jpena|off is now known as jpena07:37
*** ysandeep is now known as ysandeep|lunch09:05
opendevreviewMerged openstack/diskimage-builder master: rockylinux : create machine-id early  https://review.opendev.org/c/openstack/diskimage-builder/+/85168709:27
*** ysandeep|lunch is now known as ysandeep10:21
*** tosky_ is now known as tosky10:24
*** rlandy|out is now known as rlandy10:30
*** dasm|off is now known as dasm11:06
opendevreviewMerged openstack/diskimage-builder master: Upgrade openEuler to 22.03 LTS  https://review.opendev.org/c/openstack/diskimage-builder/+/84868011:22
*** dviroel|out is now known as dviroel11:25
fungipython 3.10.6 is out!12:05
priteau_THanks for merging the rocky fix in DIB. Do you know when a new RL8 image would be available on opendev?12:31
*** priteau_ is now known as priteau12:38
fungipriteau_: usually ianw pushes a new tag for dib, then we need to bump the minimum in the nodepool repo and merge that so new service container images are built, and then we upgrade nodepool to run from those new containers12:39
priteauOK12:40
fungiwe can probably get through all that in the next 24 hours12:40
priteauThat would be great :)12:41
outbritoHey folks, g'day! Am I crazy or gerrit is showing all WIP changes as they also were all in merge conflict (without being actually) on the dashboard?13:08
fungioutbrito: we think that started with the upgrade to gerrit 3.5 a month or so ago. clarkb has supplied a fix upstream, i think? but in short, gerrit considers wip changes unmergeable, and so treats that the same as a merge conflict13:24
fungithe downside is that it masks any actual merge conflict with wip changes, so yes we'd like to see that solved13:25
*** frenzyfriday|rover is now known as frenzyfriday|rover|lunch13:28
opendevreviewGage Hugo proposed openstack/project-config master: End project gating for openstack-helm-addons  https://review.opendev.org/c/openstack/project-config/+/85185713:37
opendevreviewGage Hugo proposed openstack/project-config master: Retire openstack-helm-addons repository  https://review.opendev.org/c/openstack/project-config/+/85185813:37
opendevreviewGage Hugo proposed openstack/project-config master: Retire openstack-helm-addons repository  https://review.opendev.org/c/openstack/project-config/+/85185813:43
*** frenzyfriday|rover|lunch is now known as frenzyfriday|rover14:05
Clark[m]fungi: I haven't pushed a fix yet because the flag is binary for "is mergable" but is used to determine if there are merge conflicts and if the change is a WIP and I'm not sure what the best fix is. Probably to have multiple flags and completely redo the index and dashboard and that is beyond my Gerrit contribution ability. I did file a bug though14:07
*** ysandeep is now known as ysandeep|dinner14:47
fungiahh, okay, i misremembered that. thanks for clarifying!14:55
*** dviroel is now known as dviroel|lunch15:14
*** ysandeep|dinner is now known as ysandeep15:29
*** ysandeep is now known as ysandeep|out15:43
*** dviroel|lunch is now known as dviroel|16:15
*** dviroel| is now known as dviroel16:15
*** jpena is now known as jpena|off16:34
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124817:18
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124817:59
fungiianw: once you're around and settled, are you good with tagging the rocky fix in dib and pushing a nodepool requirements minimum increase for that? if so, i'm on hand to expedite review and help with builder restarts18:04
clarkbnote the builders will automatically restart when the hourly infra-prod-service-nodepool job runs (so it will happen relatively quickly on its own)18:05
opendevreviewNeil Hanlon proposed openstack/diskimage-builder master: Add Rockylinux 9 build configuration and update jobs for 8 and 9  https://review.opendev.org/c/openstack/diskimage-builder/+/84890118:07
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124818:36
outbritofungi, ack18:48
fungiclarkb: oh, good point, so we just need the tag and nodepool change merged18:52
ianwthere was a request to have 848901  in the release19:06
fungik19:06
fungii'll check that out too19:07
*** kopecmartin_ is now known as kopecmartin19:09
NeilHanlonty fungi19:09
funginp, just trying to grease the wheels where i can19:10
NeilHanlonand ianw! 19:10
ianw# ls /boot19:22
ianwinitramfs-5.14.0-70.17.1.el9_0.x86_64.img19:22
ianwNeilHanlon: in testing, just in a container, I don't see kernel-core installing anything to /boot with 9 images either with or without an /etc/machine-id :/19:22
fungikernels are for suckers19:23
NeilHanlonhrm.. I ran dib manually and see this19:23
NeilHanlonhttps://paste.opendev.org/show/b8cpIzFWPB9B8LyaLpvc/19:23
ianwyeah, https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d81/848901/3/check/dib-nodepool-functional-openstack-rockylinux-9-containerfile-src/d818ba9/nodepool/builds/test-image-0000000001.log19:24
ianwdoes show that there's a kernel in there.  but how it got there is a bit of a mystery!19:24
ianwand that job passed, so the image booted19:24
NeilHanlonhttps://drop1.neilhanlon.me/irc/uploads/46ae407b3ae31e5c/image.png 19:26
NeilHanlonexcept, as a gif..19:26
fungieverything's better as a gif19:50
fungiunless you pronounce it wrong19:50
fungiit's "gif", not "gif"19:50
NeilHanlonAgreed19:53
opendevreviewMerged openstack/project-config master: Revert "Temporarily turn on console logs for rocky in iweb"  https://review.opendev.org/c/openstack/project-config/+/85152019:55
*** dviroel is now known as dviroel|biab19:56
ianwwell this seems a different path20:11
ianw /usr/lib/kernel/install.d/90-loaderentry.install fails out with20:11
ianw[ "$KERNEL_INSTALL_LAYOUT" = "bls" ] || exit 020:12
ianw+ '[' other = bls ']'20:12
ianw"Defaults to bls if $BOOT/ENTRY-TOKEN exists, or other otherwise"20:12
ianwwell, if i "mkdir `cat /etc/machine-id` and run kernel-install ... it installs a kernel20:15
fungithis is on rocky 9?20:16
ianwyeah in a rocky 9 container at leats20:17
ianwhttps://github.com/systemd/systemd/blob/main/src/boot/bootctl.c#L17920:23
ianwnow i'm more confused than ever20:23
ianw /etc/machine-id, /etc/machine-info, /etc/kernel/entry-token, and the layout field in /etc/kernel/install.conf20:23
ianw... i mean seriously ...20:23
NeilHanlonsystemd strikes again...20:27
NeilHanloninjecting `GRUB_ENABLE_BLSCFG=true` into the grubcfg __may__ help20:28
NeilHanlonidk. this is all a mess :/20:29
fungilennart is my copilot20:30
*** dviroel|biab is now known as dviroel20:31
ianwmy main concern is that we're basically in the same situation, with something installing the kernel we don't quite understand, and just waiting for that to decide to stop installing the kernel20:33
NeilHanlonyeah, I agree with you there. I'm taking this back to the rocky releng team too to see if we can unwrap it some more20:33
ianwperhaps that is just the ground state of existence 20:33
ianwok, so if "os-prober" is installed *before* kernel-core, it seems to install the kernel20:37
ianwinterestingly, *without* /etc/machine-id being set20:37
ianwi think actually it's /usr/lib/kernel/install.d/20-grub.install20:39
NeilHanloni think it may be grub2-tools-minimal20:41
NeilHanlonwhich is a dep of osprober20:42
NeilHanlonyep, i think that's it..20:42
ianw+ cp -aT /lib/modules/5.14.0-70.17.1.el9_0.x86_64/vmlinuz /boot/vmlinuz-5.14.0-70.17.1.el9_0.x86_6420:43
ianwthe thing that is actually copping the kernel into /boot is 20-grub.install ... which is grub2-common-2.06-27.el9_0.7.noarch20:43
NeilHanlonseems these tools are required by the bits of that script which matter https://rpa.st/25HQ20:44
ianwif you rm -rf /boot/* then edit 20-grub.install to run with "bash -x"20:44
ianwthen run 20:44
ianw /bin/kernel-install --verbose add 5.14.0-70.17.1.el9_0.x86_64 /lib/modules/5.14.0-70.17.1.el9_0.x86_64/vmlinuz20:44
NeilHanlon            eval "$(grub2-get-kernel-settings)" || true -- line 104 of 20-grub.install 20:45
ianwyeah, that bit is actually inside20:46
ianwif [[ "x${GRUB_ENABLE_BLSCFG}" = "xtrue" ]] || [[ ! -f /sbin/new-kernel-pkg ]]; then20:46
NeilHanlonfwiw, that variable is set in the default grub config in rocky20:47
ianw... except i don't think the container has /etc/default/grub at this point20:48
ianwwhat owns/generates that file?20:48
NeilHanlongrub2-tools20:49
ianwinteresting, i have that installed but no /etc/default/grub20:49
ianwso basically it seems like we're *not* using BLS generating these images.  that matches with what we see in the build logs20:50
ianw11MiB /opt/dib/tmp/dib_build.OSgwAG5e/built/boot/vmlinuz-5.14.0-70.17.1.el9_0.x86_6420:50
ianwthe kernel isn't under a machine-id directory20:50
ianwi guess my question/problem is that we are not really doing this intentionally, it seems to work just by chance20:51
*** dviroel is now known as dviroel|afk20:58
NeilHanlonOkay.. so I built the container as in the containerfile for 9, and entered inside it. I installed grub2-tools-minimal, and then kernel-c20:59
NeilHanlonkernel-core*20:59
NeilHanlonI got a couple errors during the kernel-core scriptlets, but the resultant bootdir: https://paste.opendev.org/show/bI6RxokGxsoNIz11G9Db/21:00
ianwthat does look like bls layout21:01
ianwwhere did 0bfb6ce5470443588d374015a6eaa5ff come from?21:01
ianwi'm assuming not /etc/machine-id21:02
NeilHanloncorrect, not from there as it's empty21:05
NeilHanlonyou ask a good question, however I know not yet the answer :) 21:05
opendevreviewMerged openstack/diskimage-builder master: Add Rockylinux 9 build configuration and update jobs for 8 and 9  https://review.opendev.org/c/openstack/diskimage-builder/+/84890121:08
*** dasm is now known as dasm|off21:09
NeilHanlonokay. so it's actually (I think for real this time) - systemd-udevd which is letting this work. It's a dependency of the grub2-minimal-tools script. It provides other files above and beyond the 20-grub.install you found.. including 20-grubby.install and 90-loaderentry.install 21:14
NeilHanlonid comes from here ianw: bin/kernel-install:[ -z "$MACHINE_ID" ] && MACHINE_ID="$(systemd-id128 new)"21:30
ianwhuh, ok21:35
ianwi don't want to waste too much of everyones time on this, but it's probably worth distilling this into the dib source somewhere so when it breaks we have some breadcrumbs21:38
opendevreviewMerged openstack/project-config master: trigger-rtd: fix role syntax  https://review.opendev.org/c/openstack/project-config/+/85127421:38
opendevreviewMerged openstack/project-config master: linter: update some YAML whitespace issues  https://review.opendev.org/c/openstack/project-config/+/85127521:39
clarkbBLS is just a special format for entries in /boot/ but it side effects how kernels are installed in rocky? (trying to keep up)21:40
clarkbI'm going to recheck my mm3 change after putting a hold in place. I don't understand the current test failure21:49
clarkbI've noticed I still have a hold of a gerrit server that I used to debug cache behaviors. I don't need that anymore so will delete the hold21:50
ianwclarkb: BLS is the format yep that puts things in /boot and i guess the theory is uses unique identifiers to stop things overwriting each other, which is/was/sometimes based on the machine-id, but is sometimes not21:55
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124822:57
clarkbfungi: ^ I think that is getting really close to having services running22:57
*** Guest6861 is now known as diablo_rojo23:05
fungioh, yay!23:05
clarkbok yup got a +1 from zuul finally which means all the services are listening on the expected ports and their configs didn't explode23:21
clarkbThat means the next steps will be an apache front end proxy and on the backend working to configure lists and domains23:21
*** rlandy is now known as rlandy|out23:22
ianwis there a mm2->mm3 migration tool?  istr it was difficult to get the archives in there23:31
clarkbthere is a system for doing it. YOu copy files around and run a tool iirc23:35
clarkbI think once we get this far enough along in the CI system one of the steps will be to test that migration with some of our exisitng mm2 lists23:35
ianwok cool.  definitely cart before horse situation optimizing that now :)23:44
fungiyeah, i think i linked/mentioned the migration tooling in the spec23:50
fungia bunch of very large communities have migrated their lists at this point so it should be reasonably battle-tested23:52

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!