Monday, 2023-08-14

noonedeadpunkhey folks. I've got an issue with building debian 12 with DIB. Basically the issue is that systemd-networkd remains disabled, while cloud-init attempts to use at (and it seems it's default?)10:03
noonedeadpunkThough I'm not sure in what element that makes sense to add the most...10:03
noonedeadpunkOr maybe I'm just missing smth?10:03
noonedeadpunklooks like that would be appropriate place?10:10
noonedeadpunkI also wonder if that is still needed/relevant?
noonedeadpunkzigo: will ping you, as you might be aware about current state of dib and bookworm :)10:43
funginoonedeadpunk: on our test nodes, or elsewhere? i thought we installed glean instead of cloud-init on the test nodes11:23
noonedeadpunknah, not in opendev CI, internally11:26
noonedeadpunkok, well, then it explains why cloud-init fail to start networking :)11:26
noonedeadpunkI was just thinking that it's me doing smth wrong rather then a potential bug11:27
funginoonedeadpunk: might be worth bringing up in #openstack-dib in that case11:27
* noonedeadpunk didn't knew about this channel11:27
fricklerwell it's also not the most active of channels, so I don't think you missed much11:31
funginoonedeadpunk: i know there's been a lot of discussion on debian-devel recently about the intersection between systemd-networking, network-manager and network-scripts (as well as the abandonment of isc-dhcp and possible use of dhcpcd for v6 prefix delegation)11:33
fungibut also, cloud-init issues on debian sometimes get discussed on the debian-cloud mailing list11:33
noonedeadpunkso the only issue is, that sytemd-networkd is just remains disabled11:34
fungii wouldn't be surprised if the netplan migration in ubuntu has some impact on it11:34
noonedeadpunkhaving `systemctl enable systemd-networkd` at the end of does the trick 11:34
noonedeadpunkthough I'm kinda not sure how to make it work more generally and potential consequences for those who don't use networkd with dib and debian11:35
noonedeadpunkand somehow, adding same to doesn't work, while I don't understand why for now. As this is expected to run after debootstrap 11:36
noonedeadpunk(and I'm using `debian` element, not `debian-minimal`11:37
fungiright, i think that's my confusion. we use debian-minimal in opendev (so that we can try to paper over network configuration differences from every distro)11:37
noonedeadpunkwell, usually cloud-init "just works" but yeah...11:38
fungiyou might try downloading a generic debian cloud image from cdimage.d.o and compare what network service configuration it ends up with. ideally the debian (non-minimal) element would default to making something similar11:38
noonedeadpunkSo, as I said, cloud-init does it's job, but it's matter to enable networkd on image boot11:41
noonedeadpunkIt's somehow more - rhel-ish (or how to call it)? that service is not auto-enabled on installation11:42
fungiright, and i don't know why the debian element doesn't enable systemd-networkd, whether that's in favor of using network-manager or network-scripts or relying on netplan to start services...11:42
noonedeadpunkand there're no ifup/down scripts or netplan (obviously)11:42
noonedeadpunkand no network manager...11:42
noonedeadpunkBut for deb11 there were ifup/down11:42
noonedeadpunkSo this is just smth new for deb12 that just wasn't covered yet I beleive, as cloud-init simply not used11:43
noonedeadpunkalso neither of these commands are really needed for quite a while:
noonedeadpunkas cloud-init creates 90-cloud-init-users, so there're 2 files with same content, even for deb11:
fungiit's possible systemd-networkd is being installed as a dependency and cloud-init just assumes if it's present then it should be used regardless of whether it's running?11:46
noonedeadpunkyeah, I guess it's like that11:46
noonedeadpunkto be fair, there's nothing else present that could provide networking11:46
noonedeadpunkso assumption is kinda valid to me11:47
fungiyou say ifupdown isn't installed?11:53
funginor isc-dhcp-client?11:54
noonedeadpunkfungi: actually, isc-dhcp-client is installed. But ifupdown not.12:05
noonedeadpunkneither is NetworkManager12:06
noonedeadpunkBut I'm doing smth like `DIB_APT_MINIMAL_CREATE_INTERFACES=0 DIB_CLOUD_INIT_DATASOURCES="ConfigDrive, OpenStack, Ec2" DIB_RELEASE=bookworm disk-image-create -o /tmp/debian_12.raw -t raw debian vm cloud-init-datasources growroot openssh-server cloud-init`12:09
noonedeadpunkHaving default DIB_APT_MINIMAL_CREATE_INTERFACES (which is 1) results in troubles in real-life instances for debian 10/11as then ifupdown conflicts with what cloud-init does configure12:10
noonedeadpunkas interface names are ens3 and DIB_APT_MINIMAL_CREATE_INTERFACES creates eth012:10
fungii'm checking uploading a current debian-12-genericcloud-amd64 image into glance so i can test boot it for comparison12:19
SvenKieskethis sounds very wrong, the part about ifupdown conflicting (who needs that anyway?) and the reliance on something like - legacy - interface names.12:21
SvenKieskebut I also have sadly no experience building debian images with diskimage-builder12:21
noonedeadpunkWell, I think it's there for compatibility with older images12:23
fungiokay, so i grabbed the latest official debian-12-genericcloud-amd64 and booted it in vexxhost-sjc1. is installed, systemd-networkd is not (nor is ifupdown, nor network-manager)12:28
fungiisc-dhcp-client is installed12:29
fungii think netplan is launching the dhcp client directly, but i need to dig into logs12:29
noonedeadpunknetplan is actually a wrapper around networkd12:30
fungiit's a wrapper around multiple things12:30
noonedeadpunka nice way to configure it12:30
noonedeadpunkyeah, true12:30
funginetplan can also drive network-manager or directly interface with wpa_supplicant...12:30
fungiit's a porcelain layer to provide a consistent ui across a diverse set of network configurations12:31
noonedeadpunkbut you can neglect most of netplan config with systemd-networkd config which will have precedence based on the config prefix12:31
fungisure, my point is systemd-networkd isn't present by default on the official cloud images12:31
noonedeadpunkI don't see what would get it installed in dib12:32
fungithis is why i recommended comparing to official debian images, as i said there's been a ton of upheaval in network configuration there12:32
noonedeadpunkneither what would install netplan12:33
fungiyou'll probably need to look at the build logs to see what step is causing systemd-networkd to get dragged in, presumably dpkg -l will indicate it's "automatically installed" (so a dependends or recommends from something else explicitly installed)12:33
fungithe only systemd subpackages on the official cloud image are systemd-resolvd, systemd-sysv and systemd-timesyncd12:34
noonedeadpunkit's weird. it looks like it gets isntalled with systemd-sysv12:37
noonedeadpunkfungi: yup, dpkg -S /lib/systemd/system/systemd-networkd.service says it's systemd12:39
noonedeadpunkso... if the image have systemd installed, it should have networkd 12:42
SvenKieskeisn't it just in recommends? mhmm12:43
fungioh, so there's no separate systemd-networkd package12:49
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE
noonedeadpunknah, it's indeed jsut pre-build into systemd12:50
fungisystemd-networkd.service  loaded    active   running   Network Configuration12:50
noonedeadpunkso wanna that or not - you will likely have it. Though I do recall that networkd was shipped independently12:50
fungiokay, so that is what's being run12:50
noonedeadpunkSo there're kinda 2 choices - install netplan or enable networkd.12:51
fungiand it looks like netplan is included in the default images, but maybe not used by cloud-init12:53
SvenKieskeah right, networkd is bundled in systemd in debian, just recently stumbled over that, as I recall. really weird, because it's possible to separate the two:
noonedeadpunkfungi: I can try adding netplan, but I'd say it should be used by cloud-init. As in Ubuntu there's no problems with this12:56
fungiso interestingly, the cloud-init log shows it used netplan, which i guess then called out to systemd-networkd12:56
noonedeadpunkIt's just not installed in resulted dib images12:56
noonedeadpunkyup, that;'s how ubuntu does work :)12:57
fungicloud-init wrote a /etc/netplan/50-cloud-init.yaml and then called `netplan generate`12:57
noonedeadpunkcloud-init have set of priorities basically, what thing to use12:57
fungiand it set dhcp4: true, dhcp6: true12:57
noonedeadpunkand IIRC netplan is attempted before networkd12:57
noonedeadpunkon my debian 12 image it creates /etc/systemd/network/10-cloud-init-ens3.network12:58
noonedeadpunkwhich also has dhcp/dns/mac12:59
fungiso anyway, to be closer to official debian cloud images, i guess you want to include the package12:59
noonedeadpunkI personally drop netplan from ubuntu as a first step to use plain networkd... But it's not about me, I guess :D13:03
fungiright, also as i said, there was a lot of heavy debate over this on the debian-devel ml recently. not everyone agrees on what the default network experience should be, obviously13:05
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Stop mnaually creating default user for cloud-init
SvenKieskedoes the current debian default image provide working facilities to restart the default networking service without running into errors? /cynism some releases ago that even did not work, like, at all, without custom config changes.13:33
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Install for Debian Bookworm
SvenKieskeI'm sure|hope that changed at least with networkd et al. *fingers crossed*13:33
noonedeadpunksystemctl restart systemd-networkd always ran perfectly13:34
noonedeadpunkbut with netplan - it's `netplan apply`13:34
noonedeadpunkbut ifupdown was always a mess13:35
fungiand let's not even talk about network-manager13:37
noonedeadpunkyeah, this one I've started dropping even before netplan :D13:38
noonedeadpunkgenerally I kinda like simplicity of netplan, though it has some nasty feature gaps13:40
fungii find network-manager convenient on portable devices, but mainly just use its nmcli tool to join arbitrary wifi networks13:45
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Stop creating default user for cloud-init
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Stop creating default user for cloud-init
Clark[m]Does the Debian element not create an image based on the upstream images like the other non -minimal elements do?13:53
noonedeadpunkit's using debootstrap as of today13:53
noonedeadpunk(from what I see)13:53
Clark[m]Huh. Every other distro element only builds up from scratch when using -minimal iirc.13:54
Clark[m]So you either choose the normal distro element and get the upstream image + your edits or use -minimal and get exactly what dib builds up using yum or deboostrap etc13:55
noonedeadpunkjust `debian` depends on debian-minimal13:55
noonedeadpunkso it's pretty much same flow13:56
noonedeadpunk(and element itself pretty much empty)13:57
SvenKieskewell "minimal" and "netinstall" tend to mean different things to different distros. I've seen examples where netinstall was less content then "minimal"13:57
noonedeadpunkfungi: fwiw, `apt install` works with cloud-init nicely13:59
Clark[m]noonedeadpunk: you shouldn't update the dib Debian element such that people building older Debian images will get different behavior. Your changes can change things for newer images with minimal impact but please don't break people still building stretch or whatever 13:59
Clark[m]You should set the values based on the distro release no new env vars people need to know about14:00
noonedeadpunkI've pushed couple of patches, but 891323 is most important to get image networking to work. rest is more aiming to do some "celan-up"14:00
noonedeadpunkClark[m]: well. I do know that, but I am pretty much confused, as current behaviour is inconsistent.14:01
noonedeadpunkAs you have DIB_DEBIAN_ALT_INIT_PACKAGE=systemd-sysv in one place and DIB_DEBIAN_ALT_INIT_PACKAGE=sysvinit in another14:02
noonedeadpunkand they're both "default"14:02
noonedeadpunkAnd confusing part is that extra_settings are obviously not respected outside of root.d14:03
Clark[m]Dib runs scripts in a defined order. You'll have to inspect what that order is and maybe cross check against your actual build logs to see what values are used where.14:03
Clark[m]My point is more that you should avoid adding new env vars people need to manage on builds that have existed for years14:04
noonedeadpunkOk, I will re-do things then14:05
*** JasonF is now known as JayF14:07
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE
noonedeadpunkI was also kinda wondering if there're are plans to drop support for really old distros. As handing legacy forever is... well.. raising complexity and codebase too much.14:14
Clark[m]Generally dib doesn't bother with eol distro releases. But cleanup isn't so ething that happens on EOL day (I think historically dib has been happy to keep stuff around unless it interferes with new non EOL stuff)14:16
Clark[m]I want to say cleanups have manifested in bootloader/grub management because simplifying for modern distros made things easier to think about for example14:20
noonedeadpunkjust systemd is default for Debian since Jessie, which is like 8 years old by now14:20
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Stop creating default user for cloud-init
zigonoonedeadpunk: That was from *before* Bullseye was released, as a hack, it should IMO be removed.15:14
clarkbinfra-root I have confirmed that gitea 1.20.2 on my held node is not producing any access log files...16:58
clarkbI was previously under the assumption I had misconfigured something or got a path wrong but now I'm suspicous this is a regression with gitea17:01
clarkbI see '2023/08/02 23:40:16 ...g/config_provider.go:325:deprecatedSetting() [E] Deprecated config option `[log]` `ENABLE_ACCESS_LOG` present. Use `[log]` `logger.access.MODE` instead. This fallback will be/has been removed in 1.21' in the logs. I wonder if they jumped the gun on that one17:07
opendevreviewClark Boylan proposed opendev/system-config master: Update to Gitea 1.20
clarkbLets see if that works. If it does then I think their docs/release notes are wrong17:16
clarkbthis is probably a huge philosophical question and opens a whole can of worms, but apparently openai will respect robots.txt entries that tell its crawler of internet knowledge to stay away17:38
clarkbI suspect that since all we do is open source anyway we shouldn't block their crawlers. But you may have content out there that you wish to set up robots.txt records for that do block it17:38
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Support ensure-kubernetes on bookworm
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Support ensure-kubernetes on bookworm
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Support ensure-kubernetes on bookworm
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Stop testing ensure-kubernetes with crio under ubuntu bionic
clarkblooks like the access log problem with gitea is fixed. I think that is a breaking change not captured in their docs. But a simple one to work around as that new patch does20:10
fungiwhat was the workaround?20:10
clarkbfungi: you have to configure a new config option (log.logger.access.MODE = file instead of log.ENABLE_ACCESS_LOG = true)20:11
clarkbI'll bring this up in the meeting tomorrow but thinking out loud we should probably upgrade etherpad soon as long as others are happy with the results there. Then plan for a gitea upgrade after that20:11
fungioh joy20:11
fungisecret configuration reqs20:11
clarkbthey do document the new thing but marked the old thing as deprecated until 1.21 (this is 1.20 so it should still work but doesn't) and didn't capture it in the release notes for 1.20 as a breaking chnage20:12
clarkbThe openstack release is happening soonish right? now is probably a good time to start preparing a gerrit upgrade but plan to do it after the openstack release20:19
clarkbanything new to add to the meeting agenda? I removed the meetpad cert item since that got resolved and added some notes to the fedora topic since I think we can accelerate that removal now20:21
fungii dunno about "soonish" but depends on your definition20:23
fungi estimates 2023-10-04 which is just shy of 2 months20:24
clarkb feature freeze is the end of the month20:24
clarkbwhich is when we typically try to avoid making big changes20:24
clarkbfungi: looks like you were able to test on the held etherpad node? Maybe we should plan to land that tomorrow?20:48
fungiyes, lgtm, i'm good upgrading whenever20:49
clarkbone of the very old things on my todo list is cleaning up the insecure ci registry that was replaced21:30
clarkb removes the old one from our inventory so that we can delete the instance21:31
clarkbalright meeting agenda is sent22:56
*** dmellado81918 is now known as dmellado819123:17

Generated by 2.17.3 by Marius Gedminas - find it at!