| tonyb | clarkb: What do you mean about by "it doesn't respect docker.io as the unqualified domain." ? | 00:35 |
|---|---|---|
| Clark[m] | tonyb: when you reference an image like opendevorg/gerrit with docker it assumes the domain is docker.io but podman doesn't and forces you to use docker.io/opendevorg/gerrit | 01:40 |
| Clark[m] | It's a small (and probably good) difference but does mean podman isn't a drop in replacement | 01:42 |
| tonyb | Oh. I'm pretty sure there is a configuration setting for that | 02:02 |
| tonyb | clarkb: For example: https://unix.stackexchange.com/questions/701784/podman-no-longer-searches-dockerhub-error-short-name-did-not-resolve-to-an which can be managed in $HOME/.config/containers/registries.conf if setting it for the whole system is undesireable. | 02:13 |
| tonyb | clarkb: or have I missed the point? | 02:13 |
| Clark[m] | Ya I think it's fine just something we need to adjust as we sort out a move. Using specific urls is better anyway. I think that podman isn't really a drop in replacement though and I don't want to continue to perpetuate that idea. The lack of syslog logging support and broke journald support is a bigger issue for us | 02:17 |
| tonyb | That's fair. I do have in the back of my mind, and I need to confirm it, that using the unqualified search will help with some of the speculative builds. | 02:19 |
| tonyb | but yes I totally get that it isn't a drop-in replacement | 02:19 |
| Clark[m] | Using podman + the zuul-jobs container roles + a zuul-registry should make speculative builds work regardless of image name and location | 02:21 |
| Clark[m] | It is just docker that can only do this for unqualified or docker.io names | 02:21 |
| tonyb | Okay | 02:22 |
| Clark[m] | The issue there is docker only allows you to configure mirrors for docker.io but podman allows you to configure them for any domain. And the speculative images stuff relies on using the builder registry as the first mirror so the images come from there if speculatively built | 02:23 |
| tonyb | Ah okay. I see now. | 02:27 |
| tonyb | ianw: Assminug you run an OpenAFS client on a Fedora/CentOS machine. where do you get the packages from? | 03:06 |
| ianw | tonyb: there was a long-term updated copr repo that recently went missing, but https://copr.fedorainfracloud.org/coprs/mseidel/openafs/ forked it | 03:17 |
| tonyb | Thanks. That seems to only have 1 failed build. | 03:17 |
| tonyb | I guess it may be easier to just create a jammy VM | 03:18 |
| ianw | upstream builds itself into rpms that install quite easily | 03:20 |
| ianw | to be completely honest, i haven't re-setup myself locally with openafs since the other repo went away as i have just logged into our mirror-update machine | 03:21 |
| tonyb | Well that works well for some ;p | 03:21 |
| ianw | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/openafs-rpm-package-build/tasks/main.yaml#L24 | 03:22 |
| ianw | but they also ship a .src.rpm so you can skip most of that | 03:23 |
| ianw | IIRC the problem is that they don't ship src rpm's on -pre releases, which is why we had to make it from first-principles there, when there was some show-stopper bug and we needed the pre-release | 03:23 |
| ianw | https://static.opendev.org/ is basically a R/O window to it, though | 03:24 |
| tonyb | Thanks. I tried building the .src.rpm and it wasn't happy on my f38 laptop. I'll figure something out | 03:25 |
| tonyb | Oh! that's basically exactly what I needed | 03:25 |
| ianw | for later fedora's like f38 i think that kafs is the way to go; there's been quite a bit of work on it | 03:27 |
| tonyb | Thanks I didn't know about that. Really I just wanted to see what was there so now I know :) | 03:28 |
| ianw | ls /afs/openstack.org | 04:03 |
| ianw | ls: cannot open directory '/afs/openstack.org': Destination address required | 04:03 |
| ianw | ... :( it has been a while since i tried kafs, was hoping it would "just work" | 04:04 |
| ianw | kafs-check-config openstack.org shows it getting the servers from the SRV records | 04:09 |
| ianw | https://bugzilla.redhat.com/show_bug.cgi?id=2208143 ... see if we can figure it out | 04:34 |
| ianw | the holdup for switching the mirrors when i last looked was cachefs | 04:35 |
| ianw | that has undergone so much change since then, it is probably worth considering when upgrading mirror nodes | 04:35 |
| opendevreview | Michal Nasiadka proposed opendev/system-config master: reprepro: mirror Ubuntu UCA Antelope for Ubuntu Jammy https://review.opendev.org/c/opendev/system-config/+/883467 | 04:53 |
| opendevreview | Michal Nasiadka proposed opendev/system-config master: reprepro: Remove UCA releases older than Ussuri https://review.opendev.org/c/opendev/system-config/+/883468 | 04:56 |
| *** amoralej|off is now known as amoralej | 06:43 | |
| *** jpena|off is now known as jpena | 07:25 | |
| *** amoralej is now known as amoralej|lunch | 12:14 | |
| *** dhill is now known as Guest546 | 12:18 | |
| *** amoralej|lunch is now known as amoralej | 13:04 | |
| artom | Hey, I'm having not much luck with our internal Gerrit helpdesk - is deleting a merged change feasible? | 13:28 |
| artom | They pointed my at https://code.stage.engineering.redhat.com/gerrit/Documentation/rest-api-changes.html#delete-change, but there are restriction in terms of change status and ownership | 13:29 |
| artom | Note: the actual git repo no longer contains the change, as it was forced push and history was changed | 13:30 |
| fungi | artom: how you do it will depend on the gerrit vintage. since i can't reach that site i can't see what version of gerrit you're asking about | 13:34 |
| artom | 3.5.4 | 13:35 |
| fungi | the change data will be in git notes within the repository in that case, and there will be named git refs for the commits for each revision | 13:36 |
| fungi | artom: also be aware there's a gerrit matrix channel where their upstream devs hang out (it's actually mostly a discord channel but mirrored into a matrix channel which is how i connect) | 13:36 |
| genekuo | I'm looking through the contributor bootstrap etherpad again to see what I can help before the migration to quay finish | 13:37 |
| fungi | https://matrix.to/#/#gerritcodereview:matrix.org | 13:37 |
| genekuo | I saw one point: arm64 devstack (what can infra do better) | 13:37 |
| Clark[m] | artom: fungi: the best option is probably https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#delete-change | 13:37 |
| genekuo | Can someone give me more context on this? | 13:37 |
| artom | Clark[m], agreed, but the change was merged | 13:37 |
| fungi | genekuo: can you remember where you saw it? | 13:37 |
| genekuo | https://etherpad.opendev.org/p/opendev-contributor-bootstrap-202304 | 13:38 |
| artom | fungi, OK, I'll try to explain that to the helpdesk person | 13:38 |
| genekuo | It's under OpenDev Admin/Operator Tooling --> Platform | 13:38 |
| artom | I have a feeling it'll go over their heads :( | 13:38 |
| artom | Many thanks! I realize this is totally out of scope for you, so I appreciate the help | 13:38 |
| Clark[m] | genekuo: ianw likely added that. I think the idea is getting devstack to run on our arm64 nodes at all since it doesn't yet today iirc | 13:38 |
| fungi | artom: it may require skilled git surgery to clean up | 13:39 |
| fungi | artom: modern gerrit doesn't have a rdbms, and instead stores everything in git refs/notes | 13:39 |
| artom | Probably a good idea, all things considered | 13:39 |
| Clark[m] | artom: fungi: I always read that as admins can delete any change and the limits apply to normal users. But I have not tested this | 13:39 |
| artom | Clark[m], I think it would still need to be New or Abandoned | 13:40 |
| genekuo | Clark: I see, let me ask him for more details tomorrow | 13:40 |
| Clark[m] | It says and and abandoned by users with permission. Otherwise administrators. This is ambiguous. I've read it one way you the other. Can probably confirm by having your admins try it :) | 13:41 |
| fungi | yeah, if the rest api method works for merged changes then that's obviously preferable. if it doesn't then git surgery is likely needed. if it comes to that, i'd recomment checking with residents of the #gerritcodereview:matrix.org channel first in case they have better suggestions/warnings | 13:41 |
| artom | Clark[m], will do | 13:41 |
| artom | fungi, ack | 13:41 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Fix the NebulOuS tenant config https://review.opendev.org/c/openstack/project-config/+/883507 | 13:44 |
| fungi | Clark[m]: genekuo: agreed, filtering https://zuul.opendev.org/t/openstack/jobs for "arm64" turns up no devstack jobs configured for it that i can see (of course i'm assuming "arm64" would appear in the job names) | 13:44 |
| genekuo | fungi: So my current understanding is that we would like to add some testing for devstack on arm64. Is it correct? | 13:46 |
| fungi | genekuo: yeah, likely start by making a basic devstack job variant with an ubuntu-jammy-arm64 nodeset and propose it as a wip change to the openstack/devstack repo. it should be self-testing on the change so you can see what breaks | 13:48 |
| fungi | actually devstack jobs seem to use a customized nodeset to set the node names explicitly, so you'll probably want to add something like a openstack-single-node-jammy-arm64 nodeset near the top of https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml as a copy of the existing openstack-single-node-jammy nodeset but with the label switched from ubuntu-jammy to ubuntu-jammy-arm64 | 13:50 |
| fungi | genekuo: then probably create a devstack-arm64 job which has the devstack job as its parent but sets the nodeset to openstack-single-node-jammy-arm64 | 13:52 |
| fungi | you can see some of the simple jobs that inherit from devstack starting around this point in the file: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L671 | 13:53 |
| fungi | genekuo: actually, the centos one is likely very close to what yours would look like (ignore the voting and timeout overrides, mainly it's an example of overriding the nodeset): https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L707-L715 | 13:55 |
| genekuo | Thanks for the information, let me look into it | 13:56 |
| fungi | happy to help, let us know if you have any questions or get stuck | 13:56 |
| opendevreview | Merged openstack/project-config master: Fix the NebulOuS tenant config https://review.opendev.org/c/openstack/project-config/+/883507 | 14:02 |
| *** hellomello is now known as yoctozepto | 14:38 | |
| yoctozepto | morning | 14:38 |
| yoctozepto | any idea why zuul picks up no jobs to run from https://review.opendev.org/c/nebulous/component-template/+/883304 ? | 14:39 |
| yoctozepto | I get no errors now; the zuul status lists the change for a moment but it disappears soon after | 14:39 |
| * yoctozepto is puzzled | 14:39 | |
| slittle | I think we might have a configuration issue. Members of starlingx-app-harbor-core (https://review.opendev.org/admin/groups/dc653984c565c07ee495d227c98551435b63d0b5,members) don't seem to have +2 voting rights on starlingx/app-harbor (https://review.opendev.org/c/starlingx/app-harbor/+/883479) | 14:42 |
| yoctozepto | noop runs but not the container job, I guess it's misconfigured but would need to guess without getting any errors back | 14:57 |
| yoctozepto | also, it does not run noop if the potentially misconfigured job is there; mysteries, mysteries... | 14:58 |
| fungi | slittle: https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/starlingx/app-harbor.config grants it to a starlingx-harbor-core group instead of starlingx-app-harbor-core, likely a typo? | 14:59 |
| fungi | looks like starlingx-app-harbor-core can abandon, starlingx-harbor-core can vote | 15:00 |
| slittle | They all should be starlingx-app-harbor-core | 15:00 |
| fungi | then seems like that was a typo in the original change, you'll want to push a change correcting those two lines | 15:05 |
| opendevreview | Scott Little proposed openstack/project-config master: Fix voting groups of starlingx/app-harbor https://review.opendev.org/c/openstack/project-config/+/883523 | 15:05 |
| slittle1 | https://review.opendev.org/c/openstack/project-config/+/883523 | 15:05 |
| fungi | yep, already saw and approved it | 15:06 |
| yoctozepto | fungi: any ideas regarding my issue? is there something in zuul logs maybe? | 15:09 |
| Clark[m] | I'm still a few minutes away from being at my desk but file filters and secrets requiring post review runs are the common reasons jobs don't run | 15:10 |
| fungi | yoctozepto: taking a look, sorry juggling a few things at once | 15:11 |
| yoctozepto | no problem | 15:11 |
| * yoctozepto patient | 15:11 | |
| yoctozepto | Clark[m]: even if the job in check does not depend on the secret? | 15:11 |
| yoctozepto | might be a zuul quirk that will go into TIL | 15:12 |
| yoctozepto | :D | 15:12 |
| Clark[m] | There are other reasons it may be post review. But no if you don't have a secret then that reason doesn't apply | 15:12 |
| Clark[m] | I'm just calling out the two common reasons for the observed behavior | 15:12 |
| yoctozepto | ack, thanks for clarifying | 15:13 |
| mnasiadka | Clark[m]: I've read somewhere around AFS space issues - I assume adding UCA/Antelope mirroring needs to wait for some cleanup? | 15:14 |
| fungi | yoctozepto: so looking at it, 883304 is adding a test-app-build-opendev-image job to the check pipeline for nebulous/component-template and you're asking why it didn't run on the change? | 15:14 |
| Clark[m] | mnasiadka UCA tends to be relatively tiny. We can probably add it. The bigger concern is Debian Bookworm which is hundreds of GB (like 400?) | 15:17 |
| fungi | and also NeilHanlon's quesion yesterday about the possibility of adding a mirror for rocky 9 | 15:17 |
| mnasiadka | Clark[m]: that's another thing, but we can work on it in a longer timeframe - Bookworm is not officially out yet | 15:19 |
| fungi | not for another month anyway | 15:19 |
| mnasiadka | Although Debian OpenStack packaging requires Bookworm for Antelope :) | 15:20 |
| mnasiadka | (but that's a minor topic) | 15:20 |
| fungi | er, well, bookworm is closer to 3 weeks away, but still not ready just yet | 15:20 |
| fungi | https://lists.debian.org/debian-devel-announce/2023/04/msg00007.html | 15:20 |
| NeilHanlon | question re: mirroring rocky; would you be skipping debug & source repos? want to get you an accurate space range | 15:21 |
| mnasiadka | would be happy if you could merge https://review.opendev.org/c/opendev/system-config/+/883467, we can release Kolla without using the UCA mirror for now, if that's an issue though | 15:21 |
| NeilHanlon | and also probably not pulling architectures not being used.. so maybe only x86? | 15:21 |
| mnasiadka | Kolla builds aarch64 as well | 15:22 |
| fungi | NeilHanlon: probably we'd skip the same things we do for centos: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-stream-mirror-update | 15:22 |
| NeilHanlon | i think i recall some ppc64le workers at some point, though | 15:22 |
| NeilHanlon | ack fungi, thats useful, thanks! | 15:22 |
| fungi | we do exclude debug and source there, yes | 15:22 |
| yoctozepto | fungi: yeah, sorry for the delay | 15:23 |
| clarkb | ya the idea is debug and source are used extremely rarely and can be fetched from upstream with minimal impact | 15:23 |
| fungi | and ppc64le, s390x... | 15:23 |
| clarkb | the only arches we have nodes for are x86_64 and aarch64 | 15:23 |
| * NeilHanlon makes a note that it'd be good to also grab the `SIGs/9` mirror for centos stream as there's mixed content (yay confusion) | 15:23 | |
| fungi | yoctozepto: you can add the debug flag in the change and zuul will report a very verbose comment with all its decision making, let me find you the relevant docs | 15:23 |
| NeilHanlon | clarkb: yep, that makes sesnse | 15:24 |
| yoctozepto | fungi: ok, thanks | 15:24 |
| clarkb | clearing out unused fedora mirror content freed about 200GB I think | 15:24 |
| fungi | and if we drop f36 it will probably clear out around that much more? | 15:24 |
| clarkb | mnasiadka: NeilHanlon right I think what we are saying is we need help not that we can't solve these problems | 15:25 |
| clarkb | just to be clear this isn't a "no" its a "we are currently space constrained lets figure that out first" | 15:25 |
| clarkb | fungi: a little less because we cleared f37 + rawhide already but ya similar ballpark | 15:25 |
| fungi | yoctozepto: https://zuul-ci.org/docs/zuul/latest/config/project.html#attr-project.%3Cpipeline%3E.debug | 15:26 |
| NeilHanlon | clarkb: acknowledged | 15:26 |
| clarkb | step 0 is removing content we don't use which I have started with fedora. There may be other content that can be simply removed but I haven't found any yet | 15:26 |
| NeilHanlon | one thing I noticed is the rsync script is not syncing hardlinks. that could potentially save some space | 15:26 |
| NeilHanlon | if your fs supports it | 15:26 |
| mnasiadka | clarkb: I'm happy to help - I already raised a change to remove old UCA content | 15:26 |
| clarkb | NeilHanlon: I don't know if openafs supports that | 15:26 |
| mnasiadka | clarkb: which probably is not much, but still maybe something | 15:27 |
| yoctozepto | fungi: thanks, testing | 15:27 |
| clarkb | mnasiadka: ya still worth doing even if small | 15:27 |
| fungi | yoctozepto: if the output from that still isn't clear, i'm happy to try to help interpret the tea leaves | 15:27 |
| clarkb | currently all of UCA is 6.34GB | 15:27 |
| yoctozepto | fungi: same silent treatment :-( | 15:27 |
| clarkb | so ya adding another UCA release should be fine | 15:27 |
| yoctozepto | do note it does *not* run noop when these jobs are kept around | 15:28 |
| fungi | yoctozepto: goes at the project level not pipeline level i think? | 15:28 |
| yoctozepto | so to me it looks like some internal error | 15:28 |
| fungi | nevermind, it does go at the pipeline level | 15:28 |
| NeilHanlon | clarkb: yeah, i'm not super familiar w/ AFS, but a quick google seems to indicate no. so, nevermind that heh | 15:28 |
| yoctozepto | ;d | 15:28 |
| clarkb | fungi: you might need to check the scheduler logs. Its possible that loading the configs is crashing in some way that doesn't produce an error? | 15:29 |
| clarkb | NeilHanlon: is there a lot of duplicated content? THat seems odd to me given how these repos are laid out | 15:30 |
| NeilHanlon | all of the .noarch. packages are the same for every architecture. it can definitely add up | 15:31 |
| clarkb | gotcha | 15:31 |
| NeilHanlon | also our `devel` repos may contain some duplicates due to how we attempt to provide every package needed to build the OS | 15:31 |
| NeilHanlon | but also you can probably skip devel | 15:31 |
| fungi | clarkb: yoctozepto: huh, yeah the scheduler states "<Change 0x7fe8d85801d0 nebulous/component-template 883304,10> in check> is a failing item because ['it has an invalid configuration']" | 15:32 |
| fungi | i wonder why it didn't comment? | 15:32 |
| clarkb | I think there are some types of config error that happen early enough that it bails out before it can report back? | 15:32 |
| opendevreview | Merged openstack/project-config master: Fix voting groups of starlingx/app-harbor https://review.opendev.org/c/openstack/project-config/+/883523 | 15:32 |
| fungi | i'll see if i can sus it out of the log | 15:32 |
| clarkb | fungi: if you grep that lin with -C context I think ou get more info | 15:33 |
| clarkb | I want to say I've seen this happen once before and that was the key | 15:33 |
| * yoctozepto looking forward to the logs | 15:33 | |
| clarkb | yum-puppetlabs is also quite large but I think tripleo is still using that? | 15:35 |
| clarkb | 165GB for puppet packages ... | 15:35 |
| yoctozepto | yeah, they keep all versions | 15:35 |
| clarkb | thats like a whole distro :) | 15:35 |
| NeilHanlon | seriously lol | 15:37 |
| yoctozepto | puppet distro | 15:37 |
| clarkb | yoctozepto: fungi: I think it is the formatting on your encrypted item | 15:38 |
| clarkb | you need to indent that | 15:38 |
| slittle | https://review.opendev.org/c/openstack/project-config/+/883523 has merged, reloaded https://review.opendev.org/c/starlingx/app-harbor/+/883479, still no +2 | 15:38 |
| clarkb | https://review.opendev.org/c/nebulous/component-template/+/883304/10/zuul.d/jobs.yaml#7 is left aligned and it should be indented to fall under the right yaml level | 15:38 |
| yoctozepto | clarkb: (!) good catch | 15:38 |
| slittle | never mind, guess it took a few minutes to propagate | 15:39 |
| clarkb | slittle: you have to wait for the zuul job to run and apply the update. YOu can see this happen in the deploy pipeline under the openstack tenant. The job is infra-prod-manage-projects | 15:39 |
| fungi | slittle: zuul will report success for the deploy pipeline on those when it's done running the job that deploys acl changes to gerrit | 15:39 |
| clarkb | you may also need to hard refresh changes if you've previously loaded then when the old config was in effect | 15:40 |
| fungi | clarkb: yoctozepto: i failed to find the job parsing error in the scheduler log | 15:40 |
| fungi | granted i'm unsure exactly how it would have been logged | 15:40 |
| fungi | checked both schedulers | 15:41 |
| yoctozepto | clarkb, fungi: clarkb was right | 15:41 |
| clarkb | yoctozepto: side note we've discovered a flaw in those jobs you are using when used with docker. THere is no speculative container image usage | 15:41 |
| yoctozepto | that indentation was causing it | 15:41 |
| clarkb | yoctozepto: this should be a non issue with podman | 15:41 |
| yoctozepto | clarkb: we can switch to podman | 15:41 |
| yoctozepto | what exactly does it mean though? | 15:42 |
| yoctozepto | I mean, the issue | 15:42 |
| fungi | depends-on to changes which alter your images won't take the changed images into account, also similarly when you approve multiple changes so they're sequenced in the gate pipeline | 15:42 |
| clarkb | yoctozepto: for example in system-config we have container build jobs then we also have deployment testing jobs that we hope fetch the built images transparently if they are available in order to test the future state and not what is already on quay.io or dockerhub | 15:43 |
| clarkb | yoctozepto: with docker this only works if your images are hosted on docker.io (not quay.io or anywhere else) because this transparent speculative fetching relies on configuring mirrors for the image location and docker only allows you to configure a mirror for docker.io nowhere else | 15:43 |
| clarkb | podman doesn't have this limitation so is fine | 15:43 |
| fungi | that is, when you have a change that pulls and uses images depending on a change that alters the building of those images | 15:43 |
| clarkb | fungi: not just a depending change. If you have a single buildset with an image build and a test of that image you run into the same problem | 15:44 |
| clarkb | but also depends-on is affected too | 15:44 |
| fungi | er, right could be within a buildset too | 15:44 |
| clarkb | so I'm currently poking around at what it would take to migrate everything we do in opendev from docker to podman :/ | 15:45 |
| clarkb | yoctozepto: since you are only building images now the impact to you is minimal | 15:45 |
| clarkb | (you can build with buildkit enabled and I think this problem goes away) | 15:46 |
| yoctozepto | oh, I see, again, thank you very much for your insights! | 15:46 |
| yoctozepto | clarkb: we plan to pull these images back into testing on kubernetes... | 15:46 |
| yoctozepto | so it's going to be far more in there than meets the eye currently | 15:46 |
| yoctozepto | I have one colleague that is getting skilled now in gerrit, zuul and kubernetes | 15:47 |
| yoctozepto | so that I'm not the only one to pursue the holy grail of project testing ;-) | 15:47 |
| yoctozepto | * and also getting skilled in the opendev ecosystem in general | 15:48 |
| clarkb | it is really frustrating because there is a 6 year old issue open on this that had patches but they got nitpicked to death and eventually thei ssue was closed | 15:53 |
| clarkb | we were so close to this just working and it didn't happen | 15:53 |
| yoctozepto | docker bad | 15:54 |
| clarkb | I'm going to look at the wiki cert before I do anyhting else today | 15:55 |
| clarkb | fungi: ^ fyi | 15:55 |
| yoctozepto | thank you for another good debugging session, see you | 15:56 |
| fungi | thanks clarkb! | 15:57 |
| *** amoralej is now known as amoralej|off | 16:03 | |
| clarkb | the price of ssl certs has gone up despite free certs becoming common. Interesting | 16:18 |
| *** jpena is now known as jpena|off | 16:29 | |
| clarkb | fungi: ok they should be in the typical location | 16:30 |
| clarkb | fungi: I think you've manuall applied them in the past? | 16:31 |
| fungi | yeah, i can take care of that part, thanks for the renewal! | 16:50 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add podman dns plugin to ensure-podman https://review.opendev.org/c/zuul/zuul-jobs/+/883552 | 17:00 |
| clarkb | the gitea job is failing in the podman switch due to us checking certain outputs like the syslog log files. I've taken this as motivation to try and check the same things via journalctl but it seems that the only way to get the messages out in a way that shows the CONTAINER_ID field also results in encoding the message content which means we can't check for specific log messages. | 17:30 |
| clarkb | Though some log messages are not being encoded the one we were previously checking in the syslog log file is | 17:30 |
| clarkb | "MESSAGE":[83,101,114,118,101,114,32,108,105,115,116,101,110,105,110,103,32,111,110,32,58,58,32,112,111,114,116,32,50,50,50,46,13,10] is 'Server listening on :: port 222.' | 17:33 |
| clarkb | now in testing I can just check for that list of bytes, but I was hoping to come up with something human readable so that we could use tests as an example for how to interact with this as humans. Does anyone know how to convince journalctl to not encode that way? | 17:33 |
| clarkb | I do think sorting out how to interact iwth logs meaningfully as humans is going to be an important part of this transition. Worst case we can have a little utility script but that feels very wrong | 17:35 |
| clarkb | looks like if you do journalctl -g 'port 222' you get those lines back but still encoded. Thats fun | 17:41 |
| clarkb | its grepping the unencoded format and then returning encoded back to you | 17:41 |
| jrosser | i've never seen that encoded form before - how does that happen? | 17:44 |
| clarkb | jrosser: https://systemd.io/JOURNAL_EXPORT_FORMATS/#journal-json-format its one of the conditions there. I'm not sure which but pretty sure it isn't the invalid utf8 one because that is all valid ascii as far as I can tell | 17:45 |
| clarkb | note I'm specifically using the json output in order to filter on CONTAINER_ID because podman isn't properly setting the syslog tag :/ | 17:46 |
| clarkb | if podman set the syslog tag this would just be `journalctl -t docker-gitea-ssh | grep 'port 222'` | 17:47 |
| fungi | CN = wiki.openstack.org | 17:50 |
| fungi | Not After : May 18 23:59:59 2024 GMT | 17:50 |
| fungi | #status log Manually updated the SSL cert on wiki.openstack.org | 17:50 |
| opendevstatus | fungi: finished logging | 17:50 |
| clarkb | fungi: thanks lgtm from here as well | 17:51 |
| fungi | thanks, i too tested it in ff just to be sure | 17:51 |
| clarkb | wow journald is the default logging driver in podman too | 17:53 |
| clarkb | how is this the default when it just doesn't work | 17:53 |
| fungi | yes, that's what i read at least | 17:53 |
| jrosser | is CONTAINER_ID a field in the journal? | 17:53 |
| clarkb | jrosser: yes it is what podman adds when you set log-opt=label=foo. It is also supposed to set this value as a syslog tag in journald but only sets this random field that journalctl doesn't support querying on out of the box as far as I can tell | 17:54 |
| clarkb | jrosser: previously we used this same tag with docker + syslog to write our files out to disk and rotate them and have nice logs that don't disappear when you delete a container for an image upgrade or whatever | 17:55 |
| clarkb | so I'm trying to approximate that with podman + journald and hitting a wall | 17:55 |
| clarkb | specifically some way to persist and rotate logs in a human readable manner (using journalctl is fine if we can come up with an incantation). | 17:56 |
| jrosser | feel like i'm missing something obvious but `journalctl CONTAINER_ID=<foo>` doesnt help? | 17:56 |
| clarkb | I'm starting to think we'll need something like journalctl --output json --all | someutilitywewrite -t foo | 17:56 |
| clarkb | jrosser: that produces no matches. But maybe I'm not encoding things correctly? | 17:57 |
| clarkb | aha its because CONTAINER_ID and CONTAINER_TAG are different fields | 17:58 |
| clarkb | arg | 17:58 |
| clarkb | this was 90% pebcak thank you for playing the role of the rubber ducky | 17:58 |
| jrosser | you are welcome :) | 17:59 |
| jrosser | as an aside i was prodding around with `journalctl -o verbose` just to see what the fields looked like, don't have any actual podman here | 18:00 |
| clarkb | ya 90% pebkac for getting the field name wrong 10% podman because syslog tagging shoudl work then the first thing I tried would've been fine :) | 18:00 |
| opendevreview | Merged zuul/zuul-jobs master: Add podman dns plugin to ensure-podman https://review.opendev.org/c/zuul/zuul-jobs/+/883552 | 18:10 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing podman as container manager for services https://review.opendev.org/c/opendev/system-config/+/883312 | 18:15 |
| clarkb | lets see if that gives us a passing job finally and screenshots and everything | 18:15 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Force cgroupfs cgroup manager with podman on ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/883593 | 21:27 |
| ianw | genekuo: yeah, there's been a bit of prior work in arm64/devstack testing but it hasn't got that far tbh | 22:10 |
| ianw | definitely check out https://review.opendev.org/c/openstack/devstack/+/708317 and https://review.opendev.org/c/openstack/devstack/+/828639/ | 22:11 |
| ianw | the performance hasn't been there to make it practical | 22:11 |
| ianw | ... but things change | 22:12 |
| opendevreview | melanie witt proposed opendev/jeepyb master: update_bug.py: Check project along with series https://review.opendev.org/c/opendev/jeepyb/+/883597 | 23:54 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!