Thursday, 2023-05-18

tonybclarkb: What do you mean about by "it doesn't respect as the unqualified domain." ?00:35
Clark[m]tonyb: when you reference an image like opendevorg/gerrit with docker it assumes the domain is but podman doesn't and forces you to use
Clark[m]It's a small (and probably good) difference but does mean podman isn't a drop in replacement 01:42
tonybOh.  I'm pretty sure there is a configuration setting for that02:02
tonybclarkb: For example: which can be managed in $HOME/.config/containers/registries.conf if setting it for the whole system is undesireable.02:13
tonybclarkb: or have I missed the point?02:13
Clark[m]Ya I think it's fine just something we need to adjust as we sort out a move. Using specific urls is better anyway. I think that podman isn't really a drop in replacement though and I don't want to continue to perpetuate that idea. The lack of syslog logging support and broke journald support is a bigger issue for us02:17
tonybThat's fair.  I do have in the back of my mind, and I need to confirm it, that using the unqualified search will help with some of the speculative builds.02:19
tonybbut yes I totally get that it isn't a drop-in replacement02:19
Clark[m]Using podman + the zuul-jobs container roles + a zuul-registry should make speculative builds work regardless of image name and location02:21
Clark[m]It is just docker that can only do this for unqualified or names02:21
Clark[m]The issue there is docker only allows you to configure mirrors for but podman allows you to configure them for any domain. And the speculative images stuff relies on using the builder registry as the first mirror so the images come from there if speculatively built02:23
tonybAh okay.  I see now.02:27
tonybianw: Assminug you run an OpenAFS client on a Fedora/CentOS machine.  where do you get the packages from?03:06
ianwtonyb: there was a long-term updated copr repo that recently went missing, but forked it 03:17
tonybThanks. That seems to only have 1 failed build.03:17
tonybI guess it may be easier to just create a jammy VM03:18
ianwupstream builds itself into rpms that install quite easily03:20
ianwto be completely honest, i haven't re-setup myself locally with openafs since the other repo went away as i have just logged into our mirror-update machine03:21
tonybWell that works well for some ;p03:21
ianwbut they also ship a .src.rpm so you can skip most of that03:23
ianwIIRC the problem is that they don't ship src rpm's on -pre releases, which is why we had to make it from first-principles there, when there was some show-stopper bug and we needed the pre-release03:23
ianw is basically a R/O window to it, though03:24
tonybThanks.  I tried building the .src.rpm and it wasn't happy on my f38 laptop.  I'll figure something out03:25
tonybOh!  that's basically exactly what I needed03:25
ianwfor later fedora's like f38 i think that kafs is the way to go; there's been quite a bit of work on it03:27
tonybThanks I didn't know about that.  Really I just wanted to see what was there so now I know :)03:28
ianwls /afs/openstack.org04:03
ianwls: cannot open directory '/afs/': Destination address required04:03
ianw... :(  it has been a while since i tried kafs, was hoping it would "just work"04:04
ianwkafs-check-config shows it getting the servers from the SRV records04:09
ianw ... see if we can figure it out04:34
ianwthe holdup for switching the mirrors when i last looked was cachefs04:35
ianwthat has undergone so much change since then, it is probably worth considering when upgrading mirror nodes04:35
opendevreviewMichal Nasiadka proposed opendev/system-config master: reprepro: mirror Ubuntu UCA Antelope for Ubuntu Jammy
opendevreviewMichal Nasiadka proposed opendev/system-config master: reprepro: Remove UCA releases older than Ussuri
*** amoralej|off is now known as amoralej06:43
*** jpena|off is now known as jpena07:25
*** amoralej is now known as amoralej|lunch12:14
*** dhill is now known as Guest54612:18
*** amoralej|lunch is now known as amoralej13:04
artomHey, I'm having not much luck with our internal Gerrit helpdesk - is deleting a merged change feasible?13:28
artomThey pointed my at, but there are restriction in terms of change status and ownership13:29
artomNote: the actual git repo no longer contains the change, as it was forced push and history was changed13:30
fungiartom: how you do it will depend on the gerrit vintage. since i can't reach that site i can't see what version of gerrit you're asking about13:34
fungithe change data will be in git notes within the repository in that case, and there will be named git refs for the commits for each revision13:36
fungiartom: also be aware there's a gerrit matrix channel where their upstream devs hang out (it's actually mostly a discord channel but mirrored into a matrix channel which is how i connect)13:36
genekuoI'm looking through the contributor bootstrap etherpad again to see what I can help before the migration to quay finish13:37
genekuoI saw one point: arm64 devstack (what can infra do better)13:37
Clark[m]artom: fungi: the best option is probably
genekuoCan someone give me more context on this?13:37
artomClark[m], agreed, but the change was merged13:37
fungigenekuo: can you remember where you saw it?13:37
artomfungi, OK, I'll try to explain that to the helpdesk person13:38
genekuoIt's under OpenDev Admin/Operator Tooling --> Platform13:38
artomI have a feeling it'll go over their heads :(13:38
artomMany thanks! I realize this is totally out of scope for you, so I appreciate the help13:38
Clark[m]genekuo: ianw likely added that. I think the idea is getting devstack to run on our arm64 nodes at all since it doesn't yet today iirc13:38
fungiartom: it may require skilled git surgery to clean up13:39
fungiartom: modern gerrit doesn't have a rdbms, and instead stores everything in git refs/notes13:39
artomProbably a good idea, all things considered13:39
Clark[m]artom: fungi: I always read that as admins can delete any change and the limits apply to normal users. But I have not tested this13:39
artomClark[m], I think it would still need to be New or Abandoned13:40
genekuoClark: I see, let me ask him for more details tomorrow13:40
Clark[m]It says and and abandoned by users with permission. Otherwise administrators. This is ambiguous. I've read it one way you the other. Can probably confirm by having your admins try it :)13:41
fungiyeah, if the rest api method works for merged changes then that's obviously preferable. if it doesn't then git surgery is likely needed. if it comes to that, i'd recomment checking with residents of the channel first in case they have better suggestions/warnings13:41
artomClark[m], will do13:41
artomfungi, ack13:41
opendevreviewRadosÅ‚aw Piliszek proposed openstack/project-config master: Fix the NebulOuS tenant config
fungiClark[m]: genekuo: agreed, filtering for "arm64" turns up no devstack jobs configured for it that i can see (of course i'm assuming "arm64" would appear in the job names)13:44
genekuofungi: So my current understanding is that we would like to add some testing for devstack on arm64. Is it correct?13:46
fungigenekuo: yeah, likely start by making a basic devstack job variant with an ubuntu-jammy-arm64 nodeset and propose it as a wip change to the openstack/devstack repo. it should be self-testing on the change so you can see what breaks13:48
fungiactually devstack jobs seem to use a customized nodeset to set the node names explicitly, so you'll probably want to add something like a openstack-single-node-jammy-arm64 nodeset near the top of as a copy of the existing openstack-single-node-jammy nodeset but with the label switched from ubuntu-jammy to ubuntu-jammy-arm6413:50
fungigenekuo: then probably create a devstack-arm64 job which has the devstack job as its parent but sets the nodeset to openstack-single-node-jammy-arm6413:52
fungiyou can see some of the simple jobs that inherit from devstack starting around this point in the file:
fungigenekuo: actually, the centos one is likely very close to what yours would look like (ignore the voting and timeout overrides, mainly it's an example of overriding the nodeset):
genekuoThanks for the information, let me look into it13:56
fungihappy to help, let us know if you have any questions or get stuck13:56
opendevreviewMerged openstack/project-config master: Fix the NebulOuS tenant config
*** hellomello is now known as yoctozepto14:38
yoctozeptoany idea why zuul picks up no jobs to run from ?14:39
yoctozeptoI get no errors now; the zuul status lists the change for a moment but it disappears soon after14:39
* yoctozepto is puzzled14:39
slittleI think we might have a configuration issue.  Members of starlingx-app-harbor-core (,members) don't seem to have +2 voting rights on starlingx/app-harbor (
yoctozeptonoop runs but not the container job, I guess it's misconfigured but would need to guess without getting any errors back14:57
yoctozeptoalso, it does not run noop if the potentially misconfigured job is there; mysteries, mysteries...14:58
fungislittle: grants it to a starlingx-harbor-core group instead of starlingx-app-harbor-core, likely a typo?14:59
fungilooks like starlingx-app-harbor-core can abandon, starlingx-harbor-core can vote15:00
slittleThey all should be starlingx-app-harbor-core15:00
fungithen seems like that was a typo in the original change, you'll want to push a change correcting those two lines15:05
opendevreviewScott Little proposed openstack/project-config master: Fix voting groups of starlingx/app-harbor
fungiyep, already saw and approved it15:06
yoctozeptofungi: any ideas regarding my issue? is there something in zuul logs maybe?15:09
Clark[m]I'm still a few minutes away from being at my desk but file filters and secrets requiring post review runs are the common reasons jobs don't run15:10
fungiyoctozepto: taking a look, sorry juggling a few things at once15:11
yoctozeptono problem15:11
* yoctozepto patient15:11
yoctozeptoClark[m]: even if the job in check does not depend on the secret?15:11
yoctozeptomight be a zuul quirk that will go into TIL15:12
Clark[m]There are other reasons it may be post review. But no if you don't have a secret then that reason doesn't apply15:12
Clark[m]I'm just calling out the two common reasons for the observed behavior 15:12
yoctozeptoack, thanks for clarifying15:13
mnasiadkaClark[m]: I've read somewhere around AFS space issues - I assume adding UCA/Antelope mirroring needs to wait for some cleanup?15:14
fungiyoctozepto: so looking at it, 883304 is adding a test-app-build-opendev-image job to the check pipeline for nebulous/component-template and you're asking why it didn't run on the change?15:14
Clark[m]mnasiadka UCA tends to be relatively tiny. We can probably add it. The bigger concern is Debian Bookworm which is hundreds of GB (like 400?)15:17
fungiand also NeilHanlon's quesion yesterday about the possibility of adding a mirror for rocky 915:17
mnasiadkaClark[m]: that's another thing, but we can work on it in a longer timeframe - Bookworm is not officially out yet15:19
funginot for another month anyway15:19
mnasiadkaAlthough Debian OpenStack packaging requires Bookworm for Antelope :)15:20
mnasiadka(but that's a minor topic)15:20
fungier, well, bookworm is closer to 3 weeks away, but still not ready just yet15:20
NeilHanlonquestion re: mirroring rocky; would you be skipping debug & source repos? want to get you an accurate space range15:21
mnasiadkawould be happy if you could merge, we can release Kolla without using the UCA mirror for now, if that's an issue though15:21
NeilHanlonand also probably not pulling architectures not being used.. so maybe only x86? 15:21
mnasiadkaKolla builds aarch64 as well15:22
fungiNeilHanlon: probably we'd skip the same things we do for centos:
NeilHanloni think i recall some ppc64le workers at some point, though15:22
NeilHanlonack fungi, thats useful, thanks!15:22
fungiwe do exclude debug and source there, yes15:22
yoctozeptofungi: yeah, sorry for the delay15:23
clarkbya the idea is debug and source are used extremely rarely and can be fetched from upstream with minimal impact15:23
fungiand ppc64le, s390x...15:23
clarkbthe only arches we have nodes for are x86_64 and aarch6415:23
* NeilHanlon makes a note that it'd be good to also grab the `SIGs/9` mirror for centos stream as there's mixed content (yay confusion)15:23
fungiyoctozepto: you can add the debug flag in the change and zuul will report a very verbose comment with all its decision making, let me find you the relevant docs15:23
NeilHanlonclarkb: yep, that makes sesnse15:24
yoctozeptofungi: ok, thanks15:24
clarkbclearing out unused fedora mirror content freed about 200GB I think15:24
fungiand if we drop f36 it will probably clear out around that much more?15:24
clarkbmnasiadka: NeilHanlon right I think what we are saying is we need help not that we can't solve these problems15:25
clarkbjust to be clear this isn't a "no" its a "we are currently space constrained lets figure that out first"15:25
clarkbfungi: a little less because we cleared f37 + rawhide already but ya similar ballpark15:25
NeilHanlonclarkb: acknowledged15:26
clarkbstep 0 is removing content we don't use which I have started with fedora. There may be other content that can be simply removed but I haven't found any yet15:26
NeilHanlonone thing I noticed is the rsync script is not syncing hardlinks. that could potentially save some space15:26
NeilHanlonif your fs supports it15:26
mnasiadkaclarkb: I'm happy to help - I already raised a change to remove old UCA content15:26
clarkbNeilHanlon: I don't know if openafs supports that15:26
mnasiadkaclarkb: which probably is not much, but still maybe something15:27
yoctozeptofungi: thanks, testing15:27
clarkbmnasiadka: ya still worth doing even if small15:27
fungiyoctozepto: if the output from that still isn't clear, i'm happy to try to help interpret the tea leaves15:27
clarkbcurrently all of UCA is 6.34GB15:27
yoctozeptofungi: same silent treatment :-(15:27
clarkbso ya adding another UCA release should be fine15:27
yoctozeptodo note it does *not* run noop when these jobs are kept around15:28
fungiyoctozepto: goes at the project level not pipeline level i think?15:28
yoctozeptoso to me it looks like some internal error15:28
funginevermind, it does go at the pipeline level15:28
NeilHanlonclarkb: yeah, i'm not super familiar w/ AFS, but a quick google seems to indicate no. so, nevermind that heh15:28
clarkbfungi: you might need to check the scheduler logs. Its possible that loading the configs is crashing in some way that doesn't produce an error?15:29
clarkbNeilHanlon: is there a lot of duplicated content? THat seems odd to me given how these repos are laid out15:30
NeilHanlonall of the .noarch. packages are the same for every architecture. it can definitely add up15:31
NeilHanlonalso our `devel` repos may contain some duplicates due to how we attempt to provide every package needed to build the OS15:31
NeilHanlonbut also you can probably skip devel15:31
fungiclarkb: yoctozepto: huh, yeah the scheduler states "<Change 0x7fe8d85801d0 nebulous/component-template 883304,10> in check> is a failing item because ['it has an invalid configuration']"15:32
fungii wonder why it didn't comment?15:32
clarkbI think there are some types of config error that happen early enough that it bails out before it can report back?15:32
opendevreviewMerged openstack/project-config master: Fix voting groups of starlingx/app-harbor
fungii'll see if i can sus it out of the log15:32
clarkbfungi: if you grep that lin with -C context I think ou get more info15:33
clarkbI want to say I've seen this happen once before and that was the key15:33
* yoctozepto looking forward to the logs15:33
clarkbyum-puppetlabs is also quite large but I think tripleo is still using that?15:35
clarkb165GB for puppet packages ...15:35
yoctozeptoyeah, they keep all versions15:35
clarkbthats like a whole distro :)15:35
NeilHanlonseriously lol15:37
yoctozeptopuppet distro15:37
clarkbyoctozepto: fungi: I think it is the formatting on your encrypted item15:38
clarkbyou need to indent that15:38
slittle has merged, reloaded,  still no +215:38
clarkb is left aligned and it should be indented to fall under the right yaml level15:38
yoctozeptoclarkb: (!) good catch15:38
slittlenever mind, guess it took a few minutes to propagate15:39
clarkbslittle: you have to wait for the zuul job to run and apply the update. YOu can see this happen in the deploy pipeline under the openstack tenant. The job is infra-prod-manage-projects15:39
fungislittle: zuul will report success for the deploy pipeline on those when it's done running the job that deploys acl changes to gerrit15:39
clarkbyou may also need to hard refresh changes if you've previously loaded then when the old config was in effect15:40
fungiclarkb: yoctozepto: i failed to find the job parsing error in the scheduler log15:40
fungigranted i'm unsure exactly how it would have been logged15:40
fungichecked both schedulers15:41
yoctozeptoclarkb, fungi: clarkb was right15:41
clarkbyoctozepto: side note we've discovered a flaw in those jobs you are using when used with docker. THere is no speculative container image usage15:41
yoctozeptothat indentation was causing it15:41
clarkbyoctozepto: this should be a non issue with podman15:41
yoctozeptoclarkb: we can switch to podman15:41
yoctozeptowhat exactly does it mean though?15:42
yoctozeptoI mean, the issue15:42
fungidepends-on to changes which alter your images won't take the changed images into account, also similarly when you approve multiple changes so they're sequenced in the gate pipeline15:42
clarkbyoctozepto: for example in system-config we have container build jobs then we also have deployment testing jobs that we hope fetch the built images transparently if they are available in order to test the future state and not what is already on or dockerhub15:43
clarkbyoctozepto: with docker this only works if your images are hosted on (not or anywhere else) because this transparent speculative fetching relies on configuring mirrors for the image location and docker only allows you to configure a mirror for nowhere else15:43
clarkbpodman doesn't have this limitation so is fine15:43
fungithat is, when you have a change that pulls and uses images depending on a change that alters the building of those images15:43
clarkbfungi: not just a depending change. If you have a single buildset with an image build and a test of that image you run into the same problem15:44
clarkbbut also depends-on is affected too15:44
fungier, right could be within a buildset too15:44
clarkbso I'm currently poking around at what it would take to migrate everything we do in opendev from docker to podman :/15:45
clarkbyoctozepto: since you are only building images now the impact to you is minimal15:45
clarkb(you can build with buildkit enabled and I think this problem goes away)15:46
yoctozeptooh, I see, again, thank you very much for your insights!15:46
yoctozeptoclarkb: we plan to pull these images back into testing on kubernetes...15:46
yoctozeptoso it's going to be far more in there than meets the eye currently15:46
yoctozeptoI have one colleague that is getting skilled now in gerrit, zuul and kubernetes15:47
yoctozeptoso that I'm not the only one to pursue the holy grail of project testing ;-)15:47
yoctozepto* and also getting skilled in the opendev ecosystem in general15:48
clarkbit is really frustrating because there is a 6 year old issue open on this that had patches but they got nitpicked to death and eventually thei ssue was closed15:53
clarkbwe were so close to this just working and it didn't happen15:53
yoctozeptodocker bad15:54
clarkbI'm going to look at the wiki cert before I do anyhting else today15:55
clarkbfungi: ^ fyi15:55
yoctozeptothank you for another good debugging session, see you15:56
fungithanks clarkb!15:57
*** amoralej is now known as amoralej|off16:03
clarkbthe price of ssl certs has gone up despite free certs becoming common. Interesting16:18
*** jpena is now known as jpena|off16:29
clarkbfungi: ok they should be in the typical location16:30
clarkbfungi: I think you've manuall applied them in the past?16:31
fungiyeah, i can take care of that part, thanks for the renewal!16:50
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add podman dns plugin to ensure-podman
clarkbthe gitea job is failing in the podman switch due to us checking certain outputs like the syslog log files. I've taken this as motivation to try and check the same things via journalctl but it seems that the only way to get the messages out in a way that shows the CONTAINER_ID field also results in encoding the message content which means we can't check for specific log messages.17:30
clarkbThough some log messages are not being encoded the one we were previously checking in the syslog log file is17:30
clarkb"MESSAGE":[83,101,114,118,101,114,32,108,105,115,116,101,110,105,110,103,32,111,110,32,58,58,32,112,111,114,116,32,50,50,50,46,13,10] is 'Server listening on :: port 222.'17:33
clarkbnow in testing I can just check for that list of bytes, but I was hoping to come up with something human readable so that we could use tests as an example for how to interact with this as humans. Does anyone know how to convince journalctl to not encode that way?17:33
clarkbI do think sorting out how to interact iwth logs meaningfully as humans is going to be an important part of this transition. Worst case we can have a little utility script but that feels very wrong17:35
clarkblooks like if you do journalctl -g 'port 222' you get those lines back but still encoded. Thats fun17:41
clarkbits grepping the unencoded format and then returning encoded back to you17:41
jrosseri've never seen that encoded form before - how does that happen?17:44
clarkbjrosser: its one of the conditions there. I'm not sure which but pretty sure it isn't the invalid utf8 one because that is all valid ascii as far as I can tell17:45
clarkbnote I'm specifically using the json output in order to filter on CONTAINER_ID because podman isn't properly setting the syslog tag :/17:46
clarkbif podman set the syslog tag this would just be `journalctl -t docker-gitea-ssh | grep 'port 222'`17:47
fungiCN = wiki.openstack.org17:50
fungiNot After : May 18 23:59:59 2024 GMT17:50
fungi#status log Manually updated the SSL cert on wiki.openstack.org17:50
opendevstatusfungi: finished logging17:50
clarkbfungi: thanks lgtm from here as well17:51
fungithanks, i too tested it in ff just to be sure17:51
clarkbwow journald is the default logging driver in podman too17:53
clarkbhow is this the default when it just doesn't work17:53
fungiyes, that's what i read at least17:53
jrosseris CONTAINER_ID a field in the journal?17:53
clarkbjrosser: yes it is what podman adds when you set log-opt=label=foo. It is also supposed to set this value as a syslog tag in journald but only sets this random field that journalctl doesn't support querying on out of the box as far as I can tell17:54
clarkbjrosser: previously we used this same tag with docker + syslog to write our files out to disk and rotate them and have nice logs that don't disappear when you delete a container for an image upgrade or whatever17:55
clarkbso I'm trying to approximate that with podman + journald and hitting a wall17:55
clarkbspecifically some way to persist and rotate logs in a human readable manner (using journalctl is fine if we can come up with an incantation).17:56
jrosserfeel like i'm missing something obvious but `journalctl CONTAINER_ID=<foo>` doesnt help?17:56
clarkbI'm starting to think we'll need something like journalctl --output json --all | someutilitywewrite -t foo17:56
clarkbjrosser: that produces no matches. But maybe I'm not encoding things correctly?17:57
clarkbaha its because CONTAINER_ID and CONTAINER_TAG are different fields17:58
clarkbthis was 90% pebcak thank you for playing the role of the rubber ducky17:58
jrosseryou are welcome :)17:59
jrosseras an aside i was prodding around with `journalctl -o verbose` just to see what the fields looked like, don't have any actual podman here18:00
clarkbya 90% pebkac for getting the field name wrong 10% podman because syslog tagging shoudl work then the first thing I tried would've been fine :)18:00
opendevreviewMerged zuul/zuul-jobs master: Add podman dns plugin to ensure-podman
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing podman as container manager for services
clarkblets see if that gives us a passing job finally and screenshots and everything18:15
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Force cgroupfs cgroup manager with podman on ubuntu
ianwgenekuo: yeah, there's been a bit of prior work in arm64/devstack testing but it hasn't got that far tbh22:10
ianwdefinitely check out and
ianwthe performance hasn't been there to make it practical22:11
ianw... but things change22:12
opendevreviewmelanie witt proposed opendev/jeepyb master: Check project along with series

Generated by 2.17.3 by Marius Gedminas - find it at!