| noonedeadpunk | hey folks. I've got an issue with building debian 12 with DIB. Basically the issue is that systemd-networkd remains disabled, while cloud-init attempts to use at (and it seems it's default?) | 10:03 |
|---|---|---|
| noonedeadpunk | Though I'm not sure in what element that makes sense to add the most... | 10:03 |
| noonedeadpunk | Or maybe I'm just missing smth? | 10:03 |
| noonedeadpunk | looks like that would be appropriate place? | 10:10 |
| noonedeadpunk | I also wonder if that is still needed/relevant? https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/debootstrap/cleanup.d/99-set-bullseye-version#L23 | 10:42 |
| noonedeadpunk | zigo: will ping you, as you might be aware about current state of dib and bookworm :) | 10:43 |
| fungi | noonedeadpunk: on our test nodes, or elsewhere? i thought we installed glean instead of cloud-init on the test nodes | 11:23 |
| noonedeadpunk | nah, not in opendev CI, internally | 11:26 |
| noonedeadpunk | ok, well, then it explains why cloud-init fail to start networking :) | 11:26 |
| noonedeadpunk | I was just thinking that it's me doing smth wrong rather then a potential bug | 11:27 |
| fungi | noonedeadpunk: might be worth bringing up in #openstack-dib in that case | 11:27 |
| * noonedeadpunk didn't knew about this channel | 11:27 | |
| frickler | well it's also not the most active of channels, so I don't think you missed much | 11:31 |
| fungi | noonedeadpunk: i know there's been a lot of discussion on debian-devel recently about the intersection between systemd-networking, network-manager and network-scripts (as well as the abandonment of isc-dhcp and possible use of dhcpcd for v6 prefix delegation) | 11:33 |
| fungi | but also, cloud-init issues on debian sometimes get discussed on the debian-cloud mailing list | 11:33 |
| noonedeadpunk | so the only issue is, that sytemd-networkd is just remains disabled | 11:34 |
| fungi | i wouldn't be surprised if the netplan migration in ubuntu has some impact on it | 11:34 |
| noonedeadpunk | having `systemctl enable systemd-networkd` at the end of https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/debootstrap/install.d/10-debian-networking does the trick | 11:34 |
| noonedeadpunk | though I'm kinda not sure how to make it work more generally and potential consequences for those who don't use networkd with dib and debian | 11:35 |
| noonedeadpunk | and somehow, adding same to https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/debian/install.d/10-cloud-opinions#L39 doesn't work, while I don't understand why for now. As this is expected to run after debootstrap | 11:36 |
| noonedeadpunk | (and I'm using `debian` element, not `debian-minimal` | 11:37 |
| fungi | right, i think that's my confusion. we use debian-minimal in opendev (so that we can try to paper over network configuration differences from every distro) | 11:37 |
| noonedeadpunk | well, usually cloud-init "just works" but yeah... | 11:38 |
| fungi | you might try downloading a generic debian cloud image from cdimage.d.o and compare what network service configuration it ends up with. ideally the debian (non-minimal) element would default to making something similar | 11:38 |
| noonedeadpunk | So, as I said, cloud-init does it's job, but it's matter to enable networkd on image boot | 11:41 |
| noonedeadpunk | It's somehow more - rhel-ish (or how to call it)? that service is not auto-enabled on installation | 11:42 |
| fungi | right, and i don't know why the debian element doesn't enable systemd-networkd, whether that's in favor of using network-manager or network-scripts or relying on netplan to start services... | 11:42 |
| noonedeadpunk | and there're no ifup/down scripts or netplan (obviously) | 11:42 |
| noonedeadpunk | and no network manager... | 11:42 |
| noonedeadpunk | But for deb11 there were ifup/down | 11:42 |
| noonedeadpunk | So this is just smth new for deb12 that just wasn't covered yet I beleive, as cloud-init simply not used | 11:43 |
| noonedeadpunk | also neither of these commands are really needed for quite a while: https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/debian/install.d/10-cloud-opinions#L27-L30 | 11:44 |
| noonedeadpunk | as cloud-init creates 90-cloud-init-users, so there're 2 files with same content, even for deb11: https://paste.openstack.org/show/baNqYRpxCXcvml9UlYbb/ | 11:45 |
| fungi | it's possible systemd-networkd is being installed as a dependency and cloud-init just assumes if it's present then it should be used regardless of whether it's running? | 11:46 |
| noonedeadpunk | yeah, I guess it's like that | 11:46 |
| noonedeadpunk | to be fair, there's nothing else present that could provide networking | 11:46 |
| noonedeadpunk | so assumption is kinda valid to me | 11:47 |
| fungi | you say ifupdown isn't installed? | 11:53 |
| fungi | nor isc-dhcp-client? | 11:54 |
| noonedeadpunk | fungi: actually, isc-dhcp-client is installed. But ifupdown not. | 12:05 |
| fungi | interesting | 12:05 |
| noonedeadpunk | neither is NetworkManager | 12:06 |
| noonedeadpunk | But I'm doing smth like `DIB_APT_MINIMAL_CREATE_INTERFACES=0 DIB_CLOUD_INIT_DATASOURCES="ConfigDrive, OpenStack, Ec2" DIB_RELEASE=bookworm disk-image-create -o /tmp/debian_12.raw -t raw debian vm cloud-init-datasources growroot openssh-server cloud-init` | 12:09 |
| noonedeadpunk | Having default DIB_APT_MINIMAL_CREATE_INTERFACES (which is 1) results in troubles in real-life instances for debian 10/11as then ifupdown conflicts with what cloud-init does configure | 12:10 |
| noonedeadpunk | as interface names are ens3 and DIB_APT_MINIMAL_CREATE_INTERFACES creates eth0 | 12:10 |
| fungi | i'm checking uploading a current debian-12-genericcloud-amd64 image into glance so i can test boot it for comparison | 12:19 |
| SvenKieske | this sounds very wrong, the part about ifupdown conflicting (who needs that anyway?) and the reliance on something like - legacy - interface names. | 12:21 |
| SvenKieske | but I also have sadly no experience building debian images with diskimage-builder | 12:21 |
| noonedeadpunk | Well, I think it's there for compatibility with older images | 12:23 |
| noonedeadpunk | *versions | 12:23 |
| fungi | okay, so i grabbed the latest official debian-12-genericcloud-amd64 and booted it in vexxhost-sjc1. netplan.io is installed, systemd-networkd is not (nor is ifupdown, nor network-manager) | 12:28 |
| noonedeadpunk | huh | 12:29 |
| fungi | isc-dhcp-client is installed | 12:29 |
| fungi | i think netplan is launching the dhcp client directly, but i need to dig into logs | 12:29 |
| noonedeadpunk | netplan is actually a wrapper around networkd | 12:30 |
| fungi | it's a wrapper around multiple things | 12:30 |
| noonedeadpunk | a nice way to configure it | 12:30 |
| noonedeadpunk | yeah, true | 12:30 |
| fungi | netplan can also drive network-manager or directly interface with wpa_supplicant... | 12:30 |
| fungi | it's a porcelain layer to provide a consistent ui across a diverse set of network configurations | 12:31 |
| noonedeadpunk | but you can neglect most of netplan config with systemd-networkd config which will have precedence based on the config prefix | 12:31 |
| noonedeadpunk | anyway | 12:31 |
| fungi | sure, my point is systemd-networkd isn't present by default on the official cloud images | 12:31 |
| noonedeadpunk | I don't see what would get it installed in dib | 12:32 |
| fungi | this is why i recommended comparing to official debian images, as i said there's been a ton of upheaval in network configuration there | 12:32 |
| noonedeadpunk | neither what would install netplan | 12:33 |
| fungi | you'll probably need to look at the build logs to see what step is causing systemd-networkd to get dragged in, presumably dpkg -l will indicate it's "automatically installed" (so a dependends or recommends from something else explicitly installed) | 12:33 |
| fungi | the only systemd subpackages on the official cloud image are systemd-resolvd, systemd-sysv and systemd-timesyncd | 12:34 |
| noonedeadpunk | it's weird. it looks like it gets isntalled with systemd-sysv | 12:37 |
| noonedeadpunk | fungi: yup, dpkg -S /lib/systemd/system/systemd-networkd.service says it's systemd | 12:39 |
| noonedeadpunk | so... if the image have systemd installed, it should have networkd | 12:42 |
| SvenKieske | isn't it just in recommends? mhmm | 12:43 |
| fungi | oh, so there's no separate systemd-networkd package | 12:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE https://review.opendev.org/c/openstack/diskimage-builder/+/891299 | 12:49 |
| noonedeadpunk | nah, it's indeed jsut pre-build into systemd | 12:50 |
| fungi | systemd-networkd.service loaded active running Network Configuration | 12:50 |
| noonedeadpunk | so wanna that or not - you will likely have it. Though I do recall that networkd was shipped independently | 12:50 |
| fungi | okay, so that is what's being run | 12:50 |
| noonedeadpunk | https://pasteboard.co/9n4zQOn48vP2.png | 12:51 |
| noonedeadpunk | So there're kinda 2 choices - install netplan or enable networkd. | 12:51 |
| fungi | and it looks like netplan is included in the default images, but maybe not used by cloud-init | 12:53 |
| SvenKieske | ah right, networkd is bundled in systemd in debian, just recently stumbled over that, as I recall. really weird, because it's possible to separate the two: https://github.com/systemd/systemd/blob/main/meson_options.txt#L131 | 12:54 |
| noonedeadpunk | fungi: I can try adding netplan, but I'd say it should be used by cloud-init. As in Ubuntu there's no problems with this | 12:56 |
| fungi | so interestingly, the cloud-init log shows it used netplan, which i guess then called out to systemd-networkd | 12:56 |
| noonedeadpunk | It's just not installed in resulted dib images | 12:56 |
| noonedeadpunk | yup, that;'s how ubuntu does work :) | 12:57 |
| fungi | cloud-init wrote a /etc/netplan/50-cloud-init.yaml and then called `netplan generate` | 12:57 |
| noonedeadpunk | cloud-init have set of priorities basically, what thing to use | 12:57 |
| fungi | and it set dhcp4: true, dhcp6: true | 12:57 |
| noonedeadpunk | and IIRC netplan is attempted before networkd | 12:57 |
| noonedeadpunk | on my debian 12 image it creates /etc/systemd/network/10-cloud-init-ens3.network | 12:58 |
| noonedeadpunk | which also has dhcp/dns/mac | 12:59 |
| fungi | so anyway, to be closer to official debian cloud images, i guess you want to include the netplan.io package | 12:59 |
| noonedeadpunk | I personally drop netplan from ubuntu as a first step to use plain networkd... But it's not about me, I guess :D | 13:03 |
| fungi | right, also as i said, there was a lot of heavy debate over this on the debian-devel ml recently. not everyone agrees on what the default network experience should be, obviously | 13:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Stop mnaually creating default user for cloud-init https://review.opendev.org/c/openstack/diskimage-builder/+/891322 | 13:28 |
| SvenKieske | does the current debian default image provide working facilities to restart the default networking service without running into errors? /cynism some releases ago that even did not work, like, at all, without custom config changes. | 13:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Install netplan.io for Debian Bookworm https://review.opendev.org/c/openstack/diskimage-builder/+/891323 | 13:33 |
| SvenKieske | I'm sure|hope that changed at least with networkd et al. *fingers crossed* | 13:33 |
| noonedeadpunk | systemctl restart systemd-networkd always ran perfectly | 13:34 |
| noonedeadpunk | but with netplan - it's `netplan apply` | 13:34 |
| noonedeadpunk | but ifupdown was always a mess | 13:35 |
| fungi | and let's not even talk about network-manager | 13:37 |
| noonedeadpunk | yeah, this one I've started dropping even before netplan :D | 13:38 |
| noonedeadpunk | generally I kinda like simplicity of netplan, though it has some nasty feature gaps | 13:40 |
| fungi | i find network-manager convenient on portable devices, but mainly just use its nmcli tool to join arbitrary wifi networks | 13:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE https://review.opendev.org/c/openstack/diskimage-builder/+/891299 | 13:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Stop creating default user for cloud-init https://review.opendev.org/c/openstack/diskimage-builder/+/891322 | 13:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Stop creating default user for cloud-init https://review.opendev.org/c/openstack/diskimage-builder/+/891322 | 13:51 |
| Clark[m] | Does the Debian element not create an image based on the upstream images like the other non -minimal elements do? | 13:53 |
| noonedeadpunk | it's using debootstrap as of today | 13:53 |
| noonedeadpunk | (from what I see) | 13:53 |
| Clark[m] | Huh. Every other distro element only builds up from scratch when using -minimal iirc. | 13:54 |
| Clark[m] | So you either choose the normal distro element and get the upstream image + your edits or use -minimal and get exactly what dib builds up using yum or deboostrap etc | 13:55 |
| noonedeadpunk | just `debian` depends on debian-minimal | 13:55 |
| noonedeadpunk | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/debian/element-deps#L1 | 13:55 |
| noonedeadpunk | so it's pretty much same flow | 13:56 |
| noonedeadpunk | (and element itself pretty much empty) | 13:57 |
| SvenKieske | well "minimal" and "netinstall" tend to mean different things to different distros. I've seen examples where netinstall was less content then "minimal" | 13:57 |
| noonedeadpunk | fungi: fwiw, `apt install netplan.io` works with cloud-init nicely | 13:59 |
| Clark[m] | noonedeadpunk: you shouldn't update the dib Debian element such that people building older Debian images will get different behavior. Your changes can change things for newer images with minimal impact but please don't break people still building stretch or whatever | 13:59 |
| Clark[m] | You should set the values based on the distro release no new env vars people need to know about | 14:00 |
| noonedeadpunk | I've pushed couple of patches, but 891323 is most important to get image networking to work. rest is more aiming to do some "celan-up" | 14:00 |
| noonedeadpunk | Clark[m]: well. I do know that, but I am pretty much confused, as current behaviour is inconsistent. | 14:01 |
| noonedeadpunk | As you have DIB_DEBIAN_ALT_INIT_PACKAGE=systemd-sysv in one place and DIB_DEBIAN_ALT_INIT_PACKAGE=sysvinit in another | 14:02 |
| noonedeadpunk | and they're both "default" | 14:02 |
| noonedeadpunk | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/debian/install.d/10-cloud-opinions#L24 | 14:02 |
| noonedeadpunk | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/debian-systemd/root.d/05-debian-systemd#L10 | 14:02 |
| noonedeadpunk | And confusing part is that extra_settings are obviously not respected outside of root.d | 14:03 |
| Clark[m] | Dib runs scripts in a defined order. You'll have to inspect what that order is and maybe cross check against your actual build logs to see what values are used where. | 14:03 |
| Clark[m] | My point is more that you should avoid adding new env vars people need to manage on builds that have existed for years | 14:04 |
| noonedeadpunk | Ok, I will re-do things then | 14:05 |
| *** JasonF is now known as JayF | 14:07 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE https://review.opendev.org/c/openstack/diskimage-builder/+/891299 | 14:13 |
| noonedeadpunk | I was also kinda wondering if there're are plans to drop support for really old distros. As handing legacy forever is... well.. raising complexity and codebase too much. | 14:14 |
| Clark[m] | Generally dib doesn't bother with eol distro releases. But cleanup isn't so ething that happens on EOL day (I think historically dib has been happy to keep stuff around unless it interferes with new non EOL stuff) | 14:16 |
| Clark[m] | I want to say cleanups have manifested in bootloader/grub management because simplifying for modern distros made things easier to think about for example | 14:20 |
| noonedeadpunk | just systemd is default for Debian since Jessie, which is like 8 years old by now | 14:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Stop creating default user for cloud-init https://review.opendev.org/c/openstack/diskimage-builder/+/891322 | 14:23 |
| zigo | noonedeadpunk: That was from *before* Bullseye was released, as a hack, it should IMO be removed. | 15:14 |
| clarkb | infra-root I have confirmed that gitea 1.20.2 on my held node is not producing any access log files... | 16:58 |
| fungi | hrm... | 16:59 |
| clarkb | I was previously under the assumption I had misconfigured something or got a path wrong but now I'm suspicous this is a regression with gitea | 17:01 |
| clarkb | I see '2023/08/02 23:40:16 ...g/config_provider.go:325:deprecatedSetting() [E] Deprecated config option `[log]` `ENABLE_ACCESS_LOG` present. Use `[log]` `logger.access.MODE` instead. This fallback will be/has been removed in 1.21' in the logs. I wonder if they jumped the gun on that one | 17:07 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update to Gitea 1.20 https://review.opendev.org/c/opendev/system-config/+/886993 | 17:16 |
| clarkb | Lets see if that works. If it does then I think their docs/release notes are wrong | 17:16 |
| clarkb | this is probably a huge philosophical question and opens a whole can of worms, but apparently openai will respect robots.txt entries that tell its crawler of internet knowledge to stay away | 17:38 |
| clarkb | I suspect that since all we do is open source anyway we shouldn't block their crawlers. But you may have content out there that you wish to set up robots.txt records for that do block it | 17:38 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Support ensure-kubernetes on bookworm https://review.opendev.org/c/zuul/zuul-jobs/+/891339 | 17:47 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Support ensure-kubernetes on bookworm https://review.opendev.org/c/zuul/zuul-jobs/+/891339 | 17:49 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Support ensure-kubernetes on bookworm https://review.opendev.org/c/zuul/zuul-jobs/+/891339 | 18:15 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Stop testing ensure-kubernetes with crio under ubuntu bionic https://review.opendev.org/c/zuul/zuul-jobs/+/891341 | 18:15 |
| clarkb | looks like the access log problem with gitea is fixed. I think that is a breaking change not captured in their docs. But a simple one to work around as that new patch does | 20:10 |
| fungi | what was the workaround? | 20:10 |
| clarkb | fungi: you have to configure a new config option (log.logger.access.MODE = file instead of log.ENABLE_ACCESS_LOG = true) | 20:11 |
| clarkb | I'll bring this up in the meeting tomorrow but thinking out loud we should probably upgrade etherpad soon as long as others are happy with the results there. Then plan for a gitea upgrade after that | 20:11 |
| fungi | oh joy | 20:11 |
| fungi | secret configuration reqs | 20:11 |
| clarkb | they do document the new thing but marked the old thing as deprecated until 1.21 (this is 1.20 so it should still work but doesn't) and didn't capture it in the release notes for 1.20 as a breaking chnage | 20:12 |
| clarkb | The openstack release is happening soonish right? now is probably a good time to start preparing a gerrit upgrade but plan to do it after the openstack release | 20:19 |
| clarkb | anything new to add to the meeting agenda? I removed the meetpad cert item since that got resolved and added some notes to the fedora topic since I think we can accelerate that removal now | 20:21 |
| fungi | i dunno about "soonish" but depends on your definition | 20:23 |
| fungi | https://releases.openstack.org/ estimates 2023-10-04 which is just shy of 2 months | 20:24 |
| clarkb | https://releases.openstack.org/bobcat/schedule.html feature freeze is the end of the month | 20:24 |
| clarkb | which is when we typically try to avoid making big changes | 20:24 |
| fungi | yep | 20:24 |
| fungi | sgtm | 20:25 |
| clarkb | fungi: looks like you were able to test on the held etherpad node? Maybe we should plan to land that tomorrow? | 20:48 |
| fungi | yes, lgtm, i'm good upgrading whenever | 20:49 |
| clarkb | one of the very old things on my todo list is cleaning up the insecure ci registry that was replaced | 21:30 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/887001 removes the old one from our inventory so that we can delete the instance | 21:31 |
| clarkb | alright meeting agenda is sent | 22:56 |
| fungi | thanks! | 22:58 |
| *** dmellado81918 is now known as dmellado8191 | 23:17 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
