Wednesday, 2023-05-17

* dpawlik reading back05:13
*** amoralej|off is now known as amoralej07:19
*** amoralej is now known as amoralej|lunch10:59
slittle1Please add me as first core in starlingx-app-node-feature-discovery-core.  Thanks!12:20
fungislittle1: done12:23
*** amoralej|lunch is now known as amoralej12:24
*** gboutry[m] is now known as gboutry13:12
*** d34dh0r5| is now known as d34dh0r5313:35
fungiapparently this is today's security clickbait:
clarkb this shows another podman and docker difference. podman-compose won't let you down containers that are not running15:18
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing podman as container manager for services
clarkbI'm going to put holds on the jobs for ^ to start working through the behavior differences15:21
clarkbafter cleaning up the unused fedora mirror content we are below 90% AFS capacity again \o/15:38
clarkboh maybe the errors in the podman compose output are ignored? This is all very confusing. It is writing errors to stderr but then returning 015:43
clarkbI think this is the actual issue "Error: error running container create option: invalid log driver: invalid argument",15:45
clarkbwhich is a difference but not the one I thought I was seeing15:45
clarkb this seems to confirm it is a difference and one that is not implemented15:47
fungii suppose that can be worked around, at least15:54
fungithough i'll miss being able to just look in syslog15:54
funginot the end of the world15:54
clarkbI'm going to push a patchset that swaps it to journald to start. Theoretically we can have journald write out to disk like syslog was. But waiting for nodes to be held first as I think that will be helpful for additional debugging15:57
clarkbrunning `podman-compose pull` again with no image updtes doesn't seem to produce output that is different in a meaningful way compared to the first pull. So you can't really tell if you are pulling new images or not :( this is currently used by the gitea role to determine if it needs to do a graceful service restart16:06
clarkbwe may need to do an image list, pull, then image list again and compare16:08
fungithat's probably more thorough anyway16:20
opendevreviewClark Boylan proposed opendev/system-config master: Modernize install-podman
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing podman as container manager for services
*** amoralej is now known as amoralej|off16:41
clarkb is going to be an issue with using journald. Not a catastrophic one just more difficult to use journald this way17:00
clarkbI think this is the first real issue that we can't just easily workaround but we can also probably live with it17:01
corvushow are we using journald with docker?17:08
clarkbcorvus: we are using syslog with docker which podman does not support. The podman suggested alternative is journald, but tagging does not work with journal whihc makes it more difficult to use (either via journalctl to see logs from a specific container or via journald + syslog integration to write out specific log files to disk)17:09
clarkbcorvus: the syslog + docker integration is so that we get log files for each container which is nice for persisting and rotating logs as default docker logging is one giant buffer and it gets deleted when you replace a container17:10
clarkbin thise case I think we will just have to accept the deficiencies with the podman journald support and learn to filter other ways17:10
corvusugh.  i guess "conmon" is the thing to grep for to get logs from (any) container17:14
corvussince most of our systems are 1vm:1container, that's probably barely workable17:14
clarkbya and it appears to have the pid there too17:15
clarkbso you can filter by conmod[$PID] or something17:15
funginot great but i think we can work with that17:16
fungimaybe it will improve in time17:16
opendevreviewJeremy Stanley proposed openstack/project-config master: Switch from deprecated require-approval to require
fungiwe have a number of other tenants in the same situation as ^17:41
NeilHanlonheya folks, wondering what's involved in adding rocky to the infra mirrors. we have had some test failures due to outdated mirrors that I think would be good to try and squash, or at least control17:47
NeilHanlons/rocky/rocky linux/17:48
fungiNeilHanlon: aside from space requirements (we've been talking about finding things to purge to keep from adding more backing volumes and making it more fragile) we'd need a public rsync server to pull from17:49
fungialso note that our mirrors are not immune to the sorts of issues you describe, we see them all the time because whatever mirror we're pulling updates from has gone stale17:50
NeilHanlonsure, sure, that makes sense. at least for the first part, I can provide access to our mirrors via rsync and make sure there's a slot available. we've been meaning to setup a better tiering system for a while, anyways..17:52
NeilHanloni can get some estimations of how much space it'd take up, though it's probably similar to CentOS 17:52
fungiusually we don't pull from the primary sources of other distros because they require that anyone doing so become an advertised part of their public mirror network, and we can't make appropriate guarantees for accessibility (nor have we okayed the likely increases in bandwidth use with our infrastructure donors)17:53
fungiso we tend to pull from tier-2 mirrors, which as you've also observed have a tendency to suffer updating issues from time to time17:54
fungialso any sort of client filtering by address would likely cause issues if we need to rebuild our mirror-update server, since we'd need to get the new addresses added to the primary's filter17:55
NeilHanloni understand, that makes sense17:57
NeilHanlonfor what it's worth, we don't have any such policy at this time. we're not 100% sure whether we will or not, though, in the future17:58
corvusi'm going to experimentally restart one of the executors; it may restart some builds.18:57
corvusi love how docker-compose pull just keeps counting up from 0% to 100% as many times as needed :)18:59
corvus#status log hard restarted ze0119:00
opendevstatuscorvus: finished logging19:00
fungiand yes, dockermath is probably a thing unto itself19:00
opendevreviewJeremy Stanley proposed openstack/project-config master: Switch from deprecated require-approval to require
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing podman as container manager for services
clarkbI think this is getting close. nodepool-builder doesn't actually talk to a zookeeper so its all unhappy at runtime. We'll need to hold the host and then probably spin all that up using the nodepool test tools19:31
clarkbI'm going to work on gerrit before doing that though since it should be more straightforward19:36
*** elodilles is now known as elodilles_ooo19:42
corvusI'm doing a hard restart of the rest of the zuul executors; this will restart jobs19:53
corvus#status log restarted remaining zuul executors19:53
opendevstatuscorvus: finished logging19:53
corvuslooks like we have like 130 nodes perpetually deleting :/19:57
clarkbare they all in rax-iad?19:58
clarkbI think it struggles19:58
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing podman as container manager for services
clarkbminor podman difference: it doesn't respect as the unqualified domain. Not a bad difference but ^ addresses that in one case21:37
clarkbI'm realizing too that we will need to sort out a transition from docker to podman somehow22:12
clarkbthe way the change is written above it jus assumes docker was never there which won't be the case in production. I'll have to think on that22:13
fungias in cleanly uninstalling docker before installing podman?22:16
fungiwe're installing from deb packages, so hopefully a purge will be thorough22:16
fungiif disruptive22:16
fungibut... also those debs were probably not up to debian (or even ubuntu) standards, so are quite likely to leave trash behind22:17
clarkbfungi: more stopping services under docker and restarting under podman22:18
clarkbwe can cleanup docker separately but services will conflict with each other (ports, bind mounts, etc) otherwise22:18
fungigot it22:24
*** dmellado95 is now known as dmellado923:04

Generated by 2.17.3 by Marius Gedminas - find it at!