Wednesday, 2022-05-04

fungioh, great point!00:07
opendevreviewMerged openstack/project-config master: Use https with apt in our ubuntu image builds
ianwfungi: so i think the idea is to use "debuild --no-sign"  then the promote/upload job should call debsign on the artifacts00:21
fungiyeah, that's what i would expect00:23
fungiwe could sign in the build and then check and replace the signature in upload, but that's pretty pointless as it either means ephemeral first keys which aren't really conveying anything or more than one key we have to manage00:24
fungithe channels between the artifact upload and download are already reasonably well-secured00:25
fungiianw: i would specifically expect `debuild -S -us`00:27
fungiassuming you only want to build a source package and not sign the result00:28
ianwi think you need -sa to upload the source tarball which ppa wants00:30
fungiisn't that what debsign will do?00:32
*** rlandy is now known as rlandy|out00:33
ianw... maybe?00:41
fungii thought -sa was "sign all" but the debuild manpage is not all that helpful00:42
ianwyeah that ends up debuild -> dpkg-buildpackage -> dpkg-genchanges00:42
fungioh, because it's a wrapper for those, yep00:42
ianwa wrapper wrapper :)00:42
fungiand dpkg-buildpackage manpage says -sa is passed along to dpkg-genchanges00:43
ianwis "pass the parcel" an american kids game?00:43
fungiso `debuild -S -sa -us` i guess?00:44
fungibuild only source package, include source package in changes file, don't sign source package00:47
fungiahh, need -uc too (don't sign changes file)00:47
ianwyeah i got "debuild -S -sa --no-sign"00:48
ianw is building the packages, i just need to figure out copying something useful back00:48
fungiand --no-sign is equivalent to -uc -us, got it00:48
*** mazzy5098812929580859495 is now known as mazzy50988129295808594900:59
ianwviola ->
fungichristine's growing a bunch of violas on our deck01:04
fungialso i had a friend in high school who played the viola01:06
fungihuh, foldoc says there was also an early hypercard-like interpreted text language out of berkeley called viola01:08
ianwmusic to my ears :)01:22
diablo_rojo_phoneLooks good to me @clarkb 01:51
*** ysandeep|sick is now known as ysandeep05:02
*** marios is now known as marios|ruck05:05
*** prometheanfire is now known as Guest006:23
opendevreviewMerged opendev/system-config master: etherpad: remove session key
opendevreviewMerged opendev/system-config master: Remove open-vm-tools from servers
*** jpena|off is now known as jpena06:59
*** pojadhav is now known as pojadhav|lunch07:38
fricklerwheel builds seem to have been going well, all wheel volumes released 3h ago07:43
*** pojadhav|lunch is now known as pojadhav07:58
*** ysandeep is now known as ysandeep|lunch08:27
frickleroh, that explains so many matrix things flapping
*** ysandeep|lunch is now known as ysandeep09:42
*** rlandy|out is now known as rlandy10:15
*** dviroel|out is now known as dviroel11:20
*** pojadhav is now known as pojadhav|brb12:04
*** ysandeep is now known as ysandeep|coffee12:22
fungi#status log Retired 11 OpenStack mailing lists which were unused for 3 years or longer:
opendevstatusfungi: finished logging13:00
*** ysandeep|coffee is now known as ysandeep13:01
*** pojadhav|brb is now known as pojadhav13:03
opendevreviewJeremy Stanley proposed openstack/project-config master: elastic-recheck: allow releasers to merge/delete
*** ysandeep is now known as ysandeep|brb13:42
jrosserjust fyi i have ubuntu-jammy jobs failing with `libsystemd-dev : Depends: libsystemd0 (= 249.11-0ubuntu3) but 249.11-0ubuntu3.1 is to be installed`13:54
Clark[m]jrosser: is that a bug in the packages? If so we're probably waiting on upstream to fix and for our mirror to sync that14:03
fricklersaw that in devstack tests, too14:03
jrosseri'm not sure actually, i was just trying to reproduce it in a VM14:03
fricklerthe pkgs were updated yesterday, from the mirror logs everything looks fine14:03
jrosserand i'm having a TIL moment with
jrosserre. 249.11-0ubuntu3.1 1 (phased 30%)14:04
frickler as example14:04
jrosseryes thats exactly whats happing in my OSA jobs too14:05
fricklerjrosser: where did you see that 30%? fun14:05
Clark[m]We should probably disable phased updates on our VMs so that they produce consistent results from one test to another. However, we may already do that due to the machine-id file? I don't recall if our VMs set that14:08
jrosseri just set this APT::Get::Always-Include-Phased-Updates "true";14:10
jrosserand i can locally update to 249.11-0ubuntu3.1 without trouble14:10
Clark[m]jrosser: if I'm reading correctly the systemd-dev package is lagging the phased libsystemd update? That definitely seems like an Ubuntu packaging bug. But I think for CI we don't want phasing and it should be disabled in our image builds14:10
fricklerykarel: ^^ fyi14:10
jrosserthere is something odd going on, as with the package held back in my VM by the phasing i didnt get an error14:11
jrosserand disabling the phasing made the install work as expected14:11
Clark[m]Ya because the -dev package wants the older version of libsystemd only14:11
Clark[m]Holding back that update makes the -dev package happy14:11
jrosseragreed that disabling the phasing is a good idea14:12
ykarelfrickler, ack Thanks14:12
Clark[m]When you allow phased updates it pulls in the phased .1 update and breaks -dev. This is almost certainly a bug in phased updates. But we don't want them anyway in CI IMO14:12
Clark[m]It is probably worth reporting the issue to Ubuntu then also updating our images. I can take a look at that after school runs if no one else has done it yet14:13
*** ysandeep|brb is now known as ysandeep14:20
fricklerI think I'm going to hold a node for the devstack job to see the exact situation on our images14:29
jrosserfrickler: i think comparing apt policy for libsystemd-dev and libsystemd0 would be interesting14:31
fricklerjrosser: yep. plus I can live test the different Phased-Updates settings14:32
fungithanks for running that down14:39
fungiin debian you see that all the time in sid (which is the main reason it's referred to as "unstable") but migration of packages to the testing suite requires those interdependencies to be satisfied and lock-step transitions like that happen all at once14:40
fungii'm surprised this is something ubuntu doesn't catch before packages get into the archive14:41
jrosserin this case i think they come from the same source package, which makes it stranger still14:41
fricklerfungi: I think the issue happens when our images install the updated lib, but phased apt decides it doesn't want the updated dev lib yet14:41
fungiyeah, but i would definitely believe that there are corner cases where they've introduced such bugs in their phased update scheme14:41
fricklerasked in #ubuntu-server now14:42
fungithat makes sense. we've also seen this before when our mirrors lag behind our image builds, though we mostly addressed it by building our images from our mirrors14:42
fricklerfungi: the thing is that this issue happens even then14:43
fungiyeah, implying it's selecting older packages than we have available on the mirror14:43
fricklerbecause the decision about the phase is different during image build than when the image is executed14:44
Clark[m]That post says no phasing in a chroot though and our builds are chroots14:44
fungiand i agree, "phased" updates is designed to result in inconsistencies, while we want consistency14:44
fricklero.k., going to test with APT::Get::Always-Include-Phased-Updates "true"; now14:45
fungiClark[m]: right, that's probably part of this. we always pull the most recent in the image build, but the vm booted from it has a 90% chance of wanting something older14:45
Clark[m]But newer should only happen if phased is enabled? Without phasing you are always older? Or is it the opposite? If opposite that seems completely broken14:47
fungireading more14:49
Clark[m]I wonder if bullseye debootstrap is clueless so we get newer rather than older without phasing14:52
fungi"otherwise phasing will be disabled, and phased updates always included."14:53
fungithat implies to me that the behavior in a chroot is to always include the new package as well14:54
fungidisabling phasing seems to imply always installing the new package, while phasing enabled adds the delay14:54
Clark[m]Wow that seems broken to me but that would explain it14:54
Clark[m]Seems like you'd rather phase opt ins in over time then flip a flag that says everyone gets the package after $time14:56
Clark[m]But that would confirm disabling phasing in our images should fix it. Initially I thought our elements should do it since consistency is a CI feature but with that behavior I think DIB should do it14:57
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Always include phased updates for Jammy
fricklerClark[m]: fungi: tested manually on if you want to have a look there ^^14:58
Clark[m]frickler: ^ maybe also push an update to dib then we can remove it from our side when dib releases with that update?14:58
fricklerah, yes, I can do that later, I was going for the quick win first ;)14:59
* frickler will have a break, feel free to update if needed14:59
opendevreviewJeremy Stanley proposed openstack/project-config master: Always include phased updates for Jammy
*** dviroel is now known as dviroel|lunch15:18
*** ysandeep is now known as ysandeep|out15:19
clarkbfungi: frickler: can we set that flag if on ubuntu and debian regardless of release? I assume older apt will ignore the config and newer apt will do the right thing?15:19
clarkbmostly concerned that with the next debuntu release we'll have to go through this again if we pin it to jammy15:20
fungii have doubts debian will make use of phased updates, but per my review comment i agree we should at least enable it for all future ubuntu versions starting with the one where they added it, but i agree it's probably safe to just set it everywhere since it should be ignored by old releases anyway15:21
clarkbdo we want to update that now or land this and do that ina  followup?15:22
clarkbI guess we can land this and then fix it in dib properly15:22
fungii would land this now and do a more thorough fix in dib instead15:22
clarkbsince dib really wants this anyway15:22
clarkb+A and then ya we can followup in dib to do more future proofing15:24
opendevreviewMerged openstack/project-config master: Always include phased updates for Jammy
fricklerclarkb: fungi: I wasn't sure whether old apt like in xenial would complain, will test before proposing it for dib15:32
fungii suppose we can force new jammy images once that's deployed?15:34
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
clarkbfungi: ya triggering new builds is probably a good idea15:35
clarkbfungi: frickler any objections to sending once my next meeting is over to announce ethercalc shutdown?15:35
fricklerclarkb: go ahead with that15:36
fricklerfirst feedback from #ubuntu-devel is that image builds should not be using phased updates15:37
fungithe implication seems to be that the newer (phased) versions always get installed if running apt in a chroot. is that a misinterpretation?15:38
clarkband dib builds in a chroot. I suppose maybe we can undo the behavior explicitly?15:39
fungialso if running apt when no machine-id is yet present in /etc (which might be the case)15:39
fungiregardless of whether the newer phased versions are a good idea to include in virtual machine base images, they're probably a good idea for ci images because we'd rather catch behavior changes from the newer phased versions early15:40
clarkbbut also we aren't using phased updates because we are in a chroot so we install the latest stuff without phasing and then break later outside of the chroot15:41
clarkbI think the ideal would be for image builds to do the most conservative thing in the general case then users can opt into newer stuff but it appears this system does not work in that manner?15:41
clarkbanyway we build in a chroot so shouldn't be phasing anything and that seems to be the root of the problem15:41
clarkbit might also make sense for the system to automatically fast forward in a phase if it already has packages from a newer phase. But I assume the mechanism here is too simple to solve for that15:44
fungibut also i wouldn't be surprised if we're initially bootstrapping packages with the older focal apt from outside the chroot, and could be installing newer packages at that time15:45
*** marios|ruck is now known as marios|out15:45
clarkbfungi: we run in a bullseye environment. But ya that could be too15:45
clarkbIn any case I agree using the latest packages consistently without variance is ideal for a CI system15:46
fungioh, good point, i forgot it's getting called from inside a container15:46
fungiand the deploy has succeeded15:47
fungiis the best way of forcing a new jammy image to nodepool dib-image-delete the oldest of the two ready ones?15:48
fungiactually we have three ready, one just came ready 16 minutes ago15:49
clarkbno that will just use the previous build instead while it builds a new image. You want the image build command instead15:49
fungino, 16 hours ago. i can't read15:49
clarkbit will replace the image just as quickly but without reverting git cache state by using an older image15:49
fungithanks! it's been submitted15:50
fungii don't see it building yet, but i guess it has to wait for an available builder15:51
fungimight be nice if dib-image-list also reported pending builds15:51
clarkbit si building on nb01 now15:51
clarkbit does15:51
clarkboh wait I see what you mean15:51
clarkba queued state ++15:51
fungiright, pending as in not yet building but needed15:52
fungianyway, i guess the delay was just the round-trip time through zk and waiting for the available builder to notice15:52
fricklerseems I messed up
frickler/opt/dib_tmp/dib_build.emPIJ8tW/hooks/root.d/60-apt-phased-updates: line 30: syntax error near unexpected token `fi'15:55
clarkbthere is a missing then on line 2615:56
fungii didn't spot that either :/15:56
clarkbunfortunate the linter can catch tab width problems but no if then else syntax issues15:56
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Fix apt-phased-updates
clarkbaha "I hope we can remove that ischroot code path in 21.10, that should give people time to change their build chroots."15:57
clarkbman I wish this was better documented (as requested in that thread and got shut down)15:57
clarkbI guess ultimately we should control this explicitly in dib elements somewhere. The changes above are still good short term workarounds15:58
fungibut also it could be that the apt we're bootstrapping with is too old to notice the phased metadata, or that we're bootstrapping before the machine-id file is present15:58
fungieither of which could also result in the observed behavior15:59
*** jpena is now known as jpena|off16:00
fricklerthere shouldn't be a machine-id file in the image IMO, because then all VMs created from it would have the same16:01
clarkb++ generally that is generated by cloud-init or similar. I'm not sure if glean does that but maybe it should16:02
fricklerfrom #ubuntu-devel I gather that all of this works kind of like expected and our override seems to be a valid solution for our CI environment16:02
jrosseri've just run into the same thing with the way OSA builds container rootfs using debootstrap16:02
jrosserbuilding a jammy rootfs on a jammy host got into exactly the same libsystemd trap16:03
fricklerit would be interesting to see how ubuntu cloud images look like, but they haven't been built for jammy since the release16:03
opendevreviewMerged openstack/project-config master: Fix apt-phased-updates
*** dviroel|lunch is now known as dviroel16:15
fricklerthis is looking better now, waiting for all the git updates to finish
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
clarkbok I'm going to send that ethercalc email to and now17:11
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
clarkbfungi: in ^ I'm trying to install debootstrap/unstable with ansible's package module. It says no package debootstrap/unstable available. Do you think that is because I'm missing an apt-get update after adding the deb repo or is that likely the package module not understanding my request to install the unstable package?17:43
clarkbI guess I can just run a shell task and do it17:44
clarkboh I need a .list extension in sources.list.d maybe that is it17:44
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
clarkbI'll use the apt module if ^ fails as it can force a cache update17:46
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
clarkbthee is a default_release flag to the apt module I'll try next if this fails17:58
fungiin theory you should see the unstable url appear in the log when apt updates its indices17:59
clarkbya but this is all behind ansible modules. I guess ultimately I can fallback to using a shell script17:59
fungithe modules are hiding the output from apt?18:00
clarkbI think so18:01
clarkbor at least summarizing it. I'm not sure if the update_cache flag will give you cache update logs18:01
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
clarkbif ^ doesn't work I may have to try the shell script option18:02
fungiyeah, i mean it should be straightforward... add unstable to the list of package repositories, update the cache of indices, specify unstable when upgrading the package. done18:04
clarkbthats what I think I've done at least18:05
fungithat said, setting default_release in the apt module may cause it to pull all debootstrap's dependencies from unstable as well, rather than just debootstrap18:06
fungiyou can tell apt "i want to install from unstable, install these things" or you can tell apt "i want to install these things and some of them are from unstable"18:06
clarkbya I wonder if my configuration of the repo isn't right somehow18:07
fungi`apt install debootstrap/unstable` is an example of the latter18:07
fungi`apt install -t unstable debootstrap` is the former18:07
clarkbthe second task in shoudl add the repo config. Then the update_cache: yes flag should pull its indexes18:08
clarkbfungi: anything obviously wrong in that second task?18:08
clarkbapt_pkg.Error: E:The value 'unstable' is invalid for APT::Default-Release as such a release is not available in the sources <- that would imply I'm doing something wrong with that setup18:09
fungiyeah, that looks right to me though18:10
fungian example line from my workstation's /etc/apt/sources.list:18:10
fungideb unstable main18:10
fungithe trailing / ideally shouldn't matter18:10
fungibut i have a feeling the update step isn't happening correctly18:11
clarkbdid I use /etc/apt/sources.list.d wrong? permissions or something?18:11
clarkbI can have apt do an explicit update first18:11
clarkbwithout trying to install anything18:11
fungiyeah, i was going to ask if you're sure update_cache does an apt update before the apt install18:12
clarkbwell the documentation implies it does but it may check the release options first18:13
fungia custom package repository i add in the .d looks similar permissions and ownership wise:18:13
fungi-rw-r--r-- 1 root root 48 Jul 14  2019 /etc/apt/sources.list.d/weather.list18:13
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
clarkbnow with a standalone apt cache update18:14
fungias a test that unstable is a recognised repository, i did `sudo apt install debootstrap/unstable` on my workstation and got:18:15
fungiSelected version '1.0.126+nmu1' (Debian:unstable [all]) for 'debootstrap'18:15
clarkbya I took this from the nodepool image builds which run shell commands18:16
clarkbso it should work, this is purely an ansible problem at this point I think. I'll probably just run shell with ansible :/18:17
fungiyes, package management is one of those things i'm not sure ansible's abstraction layer is needed for18:20
opendevreviewClark Boylan proposed opendev/system-config master: Pull keycloak from
clarkbcorvus: ^ fyi18:23
clarkbcorvus: I'm not actually sure that is correct as we may have to run keycloak:legacy to keep using wildfly instad of whatever the nwe thing is. I'm not sure if you can transition from old thing to new thing like this trivially. but testing should at least give us some info18:24
clarkbalso has a lot more info about this as we dig in18:24
clarkbfungi: ok manual refresh seems to have worked but then it failed later trying to install podman I think beacuse we may have pulled in too much stuff from unstable18:26
fungiyeah, back to shell again, just install debootstrap/unstable instead of install -t unstable debootstrap18:26
clarkbI may try to do debootstrap/unstable after the explicit cache update but then ya time to do shell after if that breaks18:27
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
clarkband of course that fails wow18:35
* clarkb breaks out the shell18:36
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
clarkbhrm that worked but the podman install failure from when we used -t unstable remains18:44
clarkbfungi: if you get a chance can you see if makes sense to you? I'm wondering if somehow unstable is being used in that podman install so we're leaking18:44
clarkbor maybe libsemanage updated in the distro and podman packaging hasn't caught up yet?18:46
clarkbhrm no, version 3.3-1 is from unstable so it certainly appears to be a leak somehow18:46
clarkboh weird libsemanage-common is explicitly installed from unstable in the nodepool docker image18:47
fungiyeah. looks like that's from "Dockerfile: explicitly install uidmap package"18:51
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
fungilooks like we got new ubuntu-jammy built 1.5 hours ago19:29
fungiseems to be uploaded everywhere now, so hopefully the package version mismatches are resolved at this point19:30
*** Guest0 is now known as prometheanfire19:47
clarkb now it tries to build jammy but fails on the zstd problem extracting a package? How does this work in our nodepool iamges?20:01
clarkboh we install zstd20:01
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib
fungito make an apple pie from scratch, you must first invent the universe20:04
opendevreviewClark Boylan proposed opendev/system-config master: Pull keycloak from
clarkbTesting shows the non wildfly modern container is not compatibile with the wildfly stuff :/ that ps updates to use the legacy tag which is wildfly whihc should hopefully work then we can think about transitioning to not wildlfy as a followup20:09
clarkbfungi: is a simple review if you have time for it20:15
fungiclarkb: graphite isn't retired, was that file just unused?20:22
Clark[m]fungi: we should be using or similar now so the host vars are not valid20:27
Clark[m]double check that though20:27
fungithe one you're deleting is inventory/service/group_vars/graphite_opendev.org20:27
fungibut yeah there's also a inventory/service/group_vars/graphite.yaml20:28
Clark[m]oh right, that was an ansible specific group that ended up getting renamed to just graphite20:28
Clark[m] there is no group anymore20:28
fungithey're identical, except that the latter adds an entry for zuul-lb20:28
fungiso i aree, this one is unused and prime for deletion20:29
opendevreviewClark Boylan proposed opendev/base-jobs master: Add ubuntu jammy x86 nodeset
opendevreviewClark Boylan proposed opendev/base-jobs master: Add Jammy Arm64 nodeset
clarkbI went ahead and split those two nodesets up into two changes because 840355 is ready to go now if I can get reviewers. But 840538 should wait on the arm64 image and node change sto land21:04
clarkbianw: if you can review the arm64 image and node changes I'll happily monitor them tomorrow after approving them too21:05
ianwclarkb: oh yeah sorry, will do21:05
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Switch py3.10 testing to Ubuntu Jammy
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Make note of python_version being a string value
clarkbhrm did the zuul v6 work break our zuul_return in testing?21:14
opendevreviewClark Boylan proposed opendev/base-jobs master: Add ubuntu jammy x86 nodeset
opendevreviewClark Boylan proposed opendev/base-jobs master: Add Jammy Arm64 nodeset
opendevreviewClark Boylan proposed opendev/base-jobs master: Fix zuul_return in ansible linter
clarkbI think that may fix it21:17
clarkbianw: also not sure if you saw but Jammy does phased package installs and by default we seem to bypass phasing in the dib image builds due to being in a chroot but then our images boot and get phased and things break. The solution appears to be to just disable phasing entirely. We've done so for infra images but I think we should probably move that into dib them remove the workaround21:28
clarkbin openstack/project/config/nodepool/elements21:28
ianwhuh, i had not heard of this21:30
fungiyeah, it was new to me as well21:31
ianwi will refrain from commenting on my first impressions...21:31
fungiseems to be an ubuntu idea which they got implemented in apt upstream, but i don't expect debian will actually use the feature21:31
clarkbalso to make things confusing disabling phasing is the same as enabling it 100% of the time rather than doing the phased rollouts21:32
clarkbprobably more accurate to say "bypass phasing and always install latest available packages"21:32
fungithe phased updates feature would have been better termed "delay updates"21:32
fungiif you disable phased updates, you're disabling the delayed rollout basically, and everything updates as it becomes available21:33
ianware we likely to see less stable updates because people are thinking "eh, if it breaks, i'll roll it back quickly"?21:35
fungithat seems to be the idea, yes21:35
fungimove fast and roll back what you break21:36
ianwAPT::Get::Never-Include-Phased-Updates would mean that you only get the updates *after* phasing?21:36
fungii think so?21:36
clarkbbut I thnk we ewrne't sure if bulseye debootstrap etc would honor it21:36
fungiper further discussion, they're really sore about people pointing out the lack of documentation there21:37
clarkbfor opendev's purposes installing latest available all the time consistently made sense for CI, but it is possible that dib needs to be more careful21:37
ianwsigh, i mean from a generic dib pov i guess it wants a flag so people can go either way with it?21:38
clarkbthat also seems reasonable21:38
ianwmaybe you want your images with "staged" updates?21:38
clarkbI think for opendev our users already expect some delta to lates tsince we build images daily21:39
clarkbwhich means if you really want latest you should upate anyway? But I'm not sure how many jobs do a system update first21:39
clarkball that to say I think opendev would be fine if we did only oldest available packages too. The key is to not mix and match whic his what pbroke stuff21:39
ianwnot many i would say, devstack probably does but generally i don't think things are running "apt-get update" before they start21:40
fungithey are because of the bindep role21:42
ianwgood point21:43
clarkbbut only for the packages in bindep right? That is probably close neough for most things21:44
fungiright, they're not doing an apt upgrade, which is maybe what ianw actually meant21:44
fungi(update pulls new indices, upgrade pulls new packages)21:45
clarkbunrelated to all this the gerrit 3.5 memory use problem has gotte nmore interesting. Uptsream thinks it may be related to a plugin in the downstream install? We may be ok after all21:46
*** dviroel is now known as dviroel|out21:50
fungiinteresting indeed. so this was observed in gerrithub then?21:53
fungiwhich plugin?21:53
clarkbgerrithub noticed it and had problems in january. Someone else is debugging it more recently in their different gerrit. They havne't narrowed it down to a specific plugin yet, but the memory leak appears to happen due to ddnyamic key values in a metrics table and gerrit core reportedly only uses static keys21:54
clarkbthe next debuggin gsteps there are to dump the keys to try and trace them back to the source21:54
fungiso no smoking gun yet21:56
clarkbinfra-root and child will add the jammy nodeset so that people can use that shorthand in their jobs. The grandchild should wait on arm4 nodes being available. That should all work now that I Fixed the zuul_return thing with zuulv621:57
ianwi just have to do some morning things, will come back to consider dib updates for staging flags, openafs builds and monitoring arm64 builds22:00
opendevreviewMerged openstack/project-config master: Add Jammy arm64 images
opendevreviewMerged opendev/base-jobs master: Fix zuul_return in ansible linter
clarkbI'm going to restart my irc client. Hopefully I come back in one piece :)22:13
clarkbI seem to have made it back22:22
clarkbbut now all my channels are in the wrong place I should reorgniaze22:23
*** rlandy is now known as rlandy|bbl22:25
clarkbfungi: any concern with approving now to add an ubuntu-jammy nodeset?22:48
fungii've approved it now, thought i'd already reviewed that22:49
clarkbIt hit the parent issue and then I also split it as arm64 instances weren't ready yet22:50
opendevreviewMerged opendev/base-jobs master: Add ubuntu jammy x86 nodeset
clarkbI just edited openstack-zuul-jobs-core to remove frickler as an explicit member (frickler is part of infra-core which is also included) and jlk as jlk is no longer active23:23
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Switch py3.10 testing to Ubuntu Jammy
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Make note of python_version being a string value
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Switch py3.10 testing to Ubuntu Jammy
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Make note of python_version being a string value

Generated by 2.17.3 by Marius Gedminas - find it at!