Thursday, 2022-12-08

clarkbI guess it isn't an error from apt-key to add an expired key00:00
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job  https://review.opendev.org/c/zuul/zuul-jobs/+/86694500:00
corvusmore allowlist fallout00:00
ianwhttps://kubic.opensuse.org/blog/2022-06-10-kubic-retired/ <- i don't know if i knew this00:02
fungiclarkb: i suppose one workaround to the --showconfig --verbose problem is to run once without --verbose and parse that, then run it again with --verbose and log it00:06
clarkbfungi: in this case we had been trying to consistently run all the tox commands using the same args so that the right envs are selected and so on. I think for showconfig it is ok for us to drop the verbose flag which is what I've done beacuse we're not interested in anything but the config00:07
clarkbits possible that we'll find scenarios where this isn't true and in that case we can possibly filter out the -vv from extra_tox_args and pass the rest in00:07
clarkbbut from a UX perspective I think a command whose purpose is to emit a config "file" should do that in a valid manner00:07
clarkbbut I'm not going to argue it further since apparently I'm complaining (I was trying to help and offered ideas to solutions and may have even tried to write a patch if one or another was selected)00:08
fungii have a feeling the maintainer may be more short than usual due to handling an influx of new v4-related issues after asking people to test prereleases00:10
fungibut yes, not worth arguing that one further, i agree00:11
clarkbthis might be a situation where a new tool would've been better00:12
fungiyes, a separate tox4 package or something00:13
clarkbif you've decided that you need a complete rewrite and backward compatibility is not feasible then tox2 is a better choice than breaking everyone00:13
clarkbthen tell everyone to switch to that or take on maintenance of the old code base00:13
fungixot00:13
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: drop kubic repos  https://review.opendev.org/c/zuul/zuul-jobs/+/86694700:14
clarkbsee also nose00:14
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: drop kubic repos on 22.04  https://review.opendev.org/c/zuul/zuul-jobs/+/86694700:15
fungiROOT: 177 E HandledError| could not recognize config file test-playbooks/python [tox/run.py:22]00:16
clarkbfungi: what is the context of that?00:17
fungithat's what 866945 is doing now00:17
fungihttps://zuul.opendev.org/t/zuul/build/7fd51570fe2e47caba067af38ed4e7f700:17
clarkbhttps://opendev.org/zuul/zuul-jobs/src/branch/master/test-playbooks/python/tox.yaml#L36 we set that flag to chnge the context of whee we are running and that dir has a tox.ini in it00:18
clarkbwe might need to set tox_config_file too?00:19
fungioh, maybe it no longer automatically looks in there for the config with v400:21
clarkbI see it00:21
clarkbwhat does tox_envlist: '%s' mean?00:22
clarkbtox -e%s ?00:22
clarkboh its an actual env why would it have that name...00:24
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default  https://review.opendev.org/c/zuul/zuul-jobs/+/86694300:25
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job  https://review.opendev.org/c/zuul/zuul-jobs/+/86694500:25
clarkbI think that will fix it00:25
ianwi guess i was mistaken, in that 22.04 has podman, but not cri-o.  it looks like ubuntu are on the microk8s train, with containerd or kata as the runtime, presumably why there's nothing about non-external cri-o packages00:27
clarkbianw: this is using minikube right?00:27
clarkbsurely they document how to run it on ubuntu? maybe the answer is "run this vm on your ubuntu"00:28
ianwyes, basically the way to install it on ubuntu is to use the suse distribution of cri-o00:28
ianw"it" == minikube00:28
ianwprobably the more "native" thing to do for ensure-kubernetes on ubuntu would be to install their mirok8s snap00:29
clarkbbut minikube is an upstream k8s project I thought. Surely they would make it work somehow? or maybe you're thinking they tie into microk8s?00:30
ianwaiui (which may be wrong) -- minikube =~ microk8s.  they both talk to container runtimes underneath -- microk8s to containerd, minikube to docker/cri-o/containerd00:32
clarkbright but they come from different places. I'm just surprised if the upstream project stopped making minikube possible on ubuntu is all00:32
clarkblike openstack dropping devstack because canonical has their deployment system for openstack00:33
ianwwell they haven't -- but you have to get cri-o from the external suse repositories00:33
ianw(the ones where the signing key has now expired)00:33
clarkbI see00:34
ianwthis is where i got a bit confused -- https://minikube.sigs.k8s.io/docs/drivers/podman/00:36
ianwyou can use podman (included in distro now) -- but *that* then talks to cri-o00:37
ianwwe already test this with the docker backend -- zuul-jobs-test-registry-buildset-registry-k8s-docker00:39
clarkbcan we use docker instead? we have to fix the shim right?00:41
ianwclarkb: heh, yes my thought too :) -> https://review.opendev.org/c/zuul/nodepool/+/86694800:48
clarkbfungi: the issue appears related to running out of the wrong dir?00:51
corvusclarkb: adding tox.ini didn't fix it, and the error message makes me think it perhaps isn't necessary ("is neither file or directory")00:51
clarkbcorvus: ya at first I thought maybe it is breaking on %s but that isn't it. If I cd into that dir and run against tox.ini renamed to foo.ini it does work00:51
clarkbso I'm making that change now00:51
clarkbwe should continue to test the use of -c. I suspect the issue is that the top level dir of that repo also has a tox.ini and it gets mad?00:52
clarkbthat might also be a bug00:52
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default  https://review.opendev.org/c/zuul/zuul-jobs/+/86694300:53
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job  https://review.opendev.org/c/zuul/zuul-jobs/+/86694500:53
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: don't install cri-o with docker backend  https://review.opendev.org/c/zuul/zuul-jobs/+/86695001:39
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default  https://review.opendev.org/c/zuul/zuul-jobs/+/86694301:49
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job  https://review.opendev.org/c/zuul/zuul-jobs/+/86694501:49
clarkbI think what is going on is that if you have a tox.ini in the current dir then tox breaks when you use -c01:50
clarkbdoesn't matter where the -c points to01:50
clarkbI think the above change works around that and also allows tox to find the tox.ini by default in the test dir for the earlier checks01:50
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: don't install cri-o with docker backend  https://review.opendev.org/c/zuul/zuul-jobs/+/86695001:55
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: don't install cri-o with docker backend  https://review.opendev.org/c/zuul/zuul-jobs/+/86695002:05
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: don't install cri-o with docker backend  https://review.opendev.org/c/zuul/zuul-jobs/+/86695002:25
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: don't install cri-o with docker backend  https://review.opendev.org/c/zuul/zuul-jobs/+/86695002:51
*** yadnesh|away is now known as yadnesh03:41
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695303:54
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695303:56
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695303:58
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695304:08
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695304:13
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695304:19
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695305:07
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695305:20
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695305:33
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695306:02
*** marios is now known as marios|ruck06:05
*** ysandeep is now known as ysandeep|afk06:20
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695307:36
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s  https://review.opendev.org/c/zuul/zuul-jobs/+/86695307:48
*** ysandeep is now known as ysandeep|lunch09:03
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: add microk8s support  https://review.opendev.org/c/zuul/zuul-jobs/+/86695309:38
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: add microk8s support  https://review.opendev.org/c/zuul/zuul-jobs/+/86695309:44
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: add microk8s support  https://review.opendev.org/c/zuul/zuul-jobs/+/86695309:52
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: add microk8s support  https://review.opendev.org/c/zuul/zuul-jobs/+/86695310:13
*** ysandeep|lunch is now known as ysandeep10:31
*** dviroel|afk is now known as dviroel10:51
*** chandankumar is now known as chkumar|afk11:00
*** rlandy|out is now known as rlandy|rover11:18
gkskHi Team 11:52
gkskI am facing one issue while connecting to nodepool vms not sure zuul user is being created or not and i am unable to ssh with ZUUL_USER_SSH_PUBLIC_KEY to nodepool vms11:55
gkskcan any one help me to resolve this issue11:55
gkskI have added public key to ZUUL_USER_SSH_PUBLIC_KEY attribute in nodepool.yaml11:55
*** pojadhav is now known as pojadhav|afk12:00
*** chkumar|afk is now known as chandankumar12:03
fungigk you need to create the user and add the executor's ssh key as authorized when building your vm images. we do it with a diskimage-builder element like https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/zuul-worker12:48
*** dasm|off is now known as dasm13:00
*** ysandeep is now known as ysandeep|afk13:11
*** pojadhav|afk is now known as pojadhav13:23
gksk@fungi I got it Thank you 13:45
*** ysandeep|afk is now known as ysandeep14:03
*** pojadhav is now known as pojadhav|dinner15:06
*** ysandeep is now known as ysandeep|out15:25
*** ykarel_ is now known as ykarel|away15:47
fungiclarkb: frickler: i guess the resolution to the # problem was to deem it notabug: "The ``#`` character now always acts as comment within ``tox.ini`` or ``setup.cfg`` tox configuration file. Where you need to pass on a ``#`` character you will need to escape it in form of ``\#`` so tox does not handle everything right of the ``#`` character as a comment." https://github.com/tox-dev/tox/pull/262116:07
*** pojadhav|dinner is now known as pojadhav16:10
*** dviroel is now known as dviroel|lunch16:10
fungiseems like any backward-incompatible behavior changes not caught and called out in prerelease testing are just going to be deemed "working as intended"16:15
clarkbfungi: yup in the original issue frickler filed they basically said not a bug so I updated my zuul-jobs stack to add \s16:18
*** marios|ruck is now known as marios|out16:39
*** dviroel|lunch is now known as dviroel17:06
opendevreviewMerged zuul/zuul-jobs master: Move tox logs into env specific dirs  https://review.opendev.org/c/zuul/zuul-jobs/+/86693917:07
*** yadnesh is now known as yadnesh|away17:16
opendevreviewMerged zuul/zuul-jobs master: Update zuul-jobs to handle tox3 and tox4  https://review.opendev.org/c/zuul/zuul-jobs/+/86693517:23
opendevreviewSorin Sb├órnea proposed zuul/zuul-jobs master: Fixed schema errors reported by ansible-lint  https://review.opendev.org/c/zuul/zuul-jobs/+/86703717:50
*** rlandy|rover is now known as rlandy|dr_appt18:39
fricklerwell in my test the \# does work as well with older tox, so I think that that's ok-ish18:47
clarkbfrickler: ya it seems to work in the chagnes I made too18:47
fungioh, good. i was worried it had to change at the same time rather than in advance18:55
clarkbjust as a heads up I'm not sure I'll have time to get to manual zuul restarts today (I didn't yesterday either). corvus indicated he thought this wasn't necessary as we have decent test coverage19:00
clarkbI've run into the tox 4 update and nodepool testing and openstacksdk stuff wall19:01
fungion pending mm3 work, we still have a few patches at topic:mailman3 if any infra-root are up for additional reviews19:02
fungialso has anyone else noticed yet that the lists.zuul-ci.org hyperkitty pages say lists.opendev.org in the top-left corner?19:02
clarkbfungi: neat its a hyperlink to the proper url though19:03
clarkbI'm guessing thats a bug if it can render the link properly but not the text19:03
fungipossible we simply missed a config option somewhere19:04
fungioh, the mm3 changes have enough votes to approve. i'll just self-approve them19:04
opendevreviewMerged opendev/zone-zuul-ci.org master: Restore the default TTL to lists  https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/86661519:07
opendevreviewMerged opendev/zone-opendev.org master: Restore the default TTL to lists  https://review.opendev.org/c/opendev/zone-opendev.org/+/86660719:07
*** dviroel_ is now known as dviroel19:17
opendevreviewMerged opendev/system-config master: Remove opendev and zuul sites from old mm2 server  https://review.opendev.org/c/opendev/system-config/+/86663019:30
*** rlandy|dr_appt is now known as rlandy|rover19:55
*** dviroel is now known as dviroel|afk20:20
*** anbanerj is now known as frenzy_friday20:44
fungiinfra-prod-base failure deploying the lists.openstack.org config cleanup: https://zuul.opendev.org/t/openstack/build/9fb5e8eaa5f449afb3e1c43d26341f9c21:01
opendevreviewClark Boylan proposed openstack/project-config master: Fix nodepool API ops Grafana graphs  https://review.opendev.org/c/openstack/project-config/+/86705421:02
clarkbinfra-root ^ I believe that will fix the nodepool api oerations graphs21:02
clarkbfungi: you need to look at the file on bridge or decrypt the one in swift21:03
clarkbthe job log itself is unfortunately less useful for infra-prod jobs21:03
fungiTASK [base/server : Install distro specific packages] fatal: [afs02.dfw.openstack.org]: FAILED! => no available installation candidate for yaml-mode21:03
fungiclarkb: yeah, that's where i was headed21:03
clarkbthat pacakge seems to exist on many ubuntus maybe apt is sad for some reason/21:04
fungii guess that's coming from here: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/base/server/vars/Debian.yaml#L421:04
clarkbya I think we install things like that to make corvus and ianw lives easier (they are emacs users iirc)21:05
fungi`apt search yaml-mode` on afs02.dfw.openstack.org says there's no installation candidate for it21:06
clarkbmight need to manually try to install the package heh way ahead of me21:06
clarkbthats weird21:06
clarkbits focal or jammy right?21:06
clarkbboth have the package but it is a virtual package21:06
fungifocal21:06
fungiNote, selecting 'elpa-yaml-mode' instead of 'yaml-mode'21:06
fungielpa-yaml-mode is already the newest version (0.0.14-1).21:07
clarkbhas newer ansible's package management broken handling of virtual packages?21:07
clarkbthat could be a side effect of updating ansible on rbdige21:07
fungithat seems likely, since `apt install yaml-mode` does nothing (noting that "elpa-yaml-mode is already the newest version") and exits 021:08
fungiwhen did we update ansible on bridge?21:09
fungiinfra-prod-base ran successfully as recently as 2022-12-06 15:37:0221:09
fungiperiodic at 2022-12-07 02:54:42 failed21:10
fungiso that's a ~11 hour window21:10
clarkbfungi: it was yesterday I think once ianw's stack of changes to address that landed21:10
fungilooking at https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-base&project=opendev/system-config21:10
clarkbianw would have a better idea of timeframe since i was busy looking at tox21:11
fungiyeah, i agree timing seems like it could line up in that case21:11
clarkbits possible we'll have a number of these small issues to sort out with new ansible :(21:11
fungiwhere "sort out" means stop using ansible modules and replace them with shell commands ansible-lint will no doubt complain about until we stop running it21:11
clarkbor use the actual package name instead? looks like bionic and older have proper yaml-mode and focal and newer use the virtual package21:12
clarkbfungi: its possible this is a bug in the package module and not in the apt module? we could try swapping those out for one another. It is curious that this doesn't error in our own ci though. Maybe when it installs it the first time it works fine?21:13
fungiyeah, possible. for now i'm trying to think how we actually test this21:13
clarkbwe do have jobs that run the base playbook. Maybe update the job to run the playbook twice back to back?21:13
clarkbwe do that with a few other playbooks (gitea off he top of my head)21:14
clarkbI think you just list the playbook twice in the list of playbooks to have the test harness run21:14
fungioh, thinking that installing virtual packages is working but upgrading them is broken?21:15
fungii guess that's a possibility21:16
clarkbyes since the ansible upgrade passed testing that implies it works on a node that doesn't have yaml-mode preinstalled on it and what we are seeing here is a node failing when it is preinstalled21:16
clarkbso I wonder if this is an idempotency bug21:16
fungireasonable assumption21:17
clarkbI think running the base playbook twice in ci is a good idempotency check either way21:19
clarkbso would be worthwhile writing that change even if it ends up being a dead end for this issue21:19
fungilooks like all the jobs inheriting from system-config-run use playbooks/zuul/run-base.yaml in the run phase21:19
fungii suppose the system-config-run-base job is a good place to do that?21:20
clarkbfungi: system-config-run-base is the job yup21:20
clarkbit has a variety of nodes to match the various distro releases21:20
fungioh, though other jobs also inherit from system-config-run-base21:20
fungibut i guess they replace the run phase when doing so21:20
clarkbthats fine they all override the nodeset and playbooks21:21
clarkbyup21:21
fungisystem-config-run-base doesn't actually have its own run playbook because that's provided by system-config-run21:21
fungiso should i double it up in system-config-run-base or in system-config-run?21:21
fungii guess if i do it in system-config-run-base then there's no point to system-config-run having a run phase itself21:22
clarkbI suspect the parent one has it because it needs a run defined? Or will zuul handle that case?21:22
fungiyeah, that's what i expect is happening21:22
clarkbI'd do it in the child (run-base) since that is the actual thing doing it and the parent is just bookkeeping21:22
fungiany preference whether i add two copies of the base playbook to the run phase of system-config-run-base and then clean up the one in system-config-run which will be unused, or add a second one to system-config-run21:23
fungiokay, so the first21:23
fungisystem-config-run is already abstract so nothing can actually run it21:24
fungioh, though system-config-run-containers is parented to system-config-run and doesn't add anything at all so i guess the run phase in system-config-run won't actually be unused21:25
ianwhrm, i feel like this elpa yaml mode is a virtual package has been a bug before ...21:25
fungialso, other jobs inherit from system-config-run-base instead of from system-config-run without setting a run playbook (system-config-run-base-ansible-devel at least) so would inherit the doubling21:26
fungiactually that's the only one, so maybe it's fine21:26
ianwhttps://github.com/ansible/ansible/issues/7677921:27
clarkbianw: do you know if `package` uses `apt` or if it is a separate codebase?21:28
clarkbin this case we appare to use package but maybe we should switch to apt?21:28
opendevreviewJeremy Stanley proposed opendev/system-config master: Run the base playbook twice to test idempotency  https://review.opendev.org/c/opendev/system-config/+/86705821:28
ianwclarkb: well that fix should be in 2.14 ...21:29
ianwi'm not sure i'm understanding how we missed it if it's broken, because we found it with our -devel job ...21:30
fungiunless we're testing with a different version of ansible than we've got on bridge?21:31
ianwwe shouldn't be ... but something feels amiss here.  i just have to do school run and can poke more21:32
*** dasm is now known as dasm|off21:50
ianw"no available installation candidate for yaml-mode" does seem different to "E: Version '0.0.14-1' for 'yaml-mode' was not found"22:08
ianwbut in the same general region i guess22:08
ianwafs01, afs02, afsdb02, lists.openstack.org & lists.katacontainers.io are the listed failures22:09
ianwthey are all focal hosts22:10
ianw /usr/ansible-venv/bin/ansible -vvv afs01.dfw.openstack.org -m package -a 'package=yaml-mode' works ...?22:15
jrosserapt history.log can be useful to find exactly what ansible attempted to do22:15
jrosserthe command never ends up being displayed22:15
ianwjrosser: yeah, good idea, unfortunately nothing referring to yaml-mode in there :/22:19
ianwalso a playbook using package: doesn't trigger it against afs0122:20
ianwi guess i'll try a manual run of the base playbook limited to the host22:21
jrosseras far as i know package: is just a proxy type thing that calls the platform specifit apt:/yum: modules22:22
ianw... which passes too ...22:26
fungithe idempotency change for the base playbook seems to also work and not hit that error, though it failed during log collection with a connection failure22:30
ianwit's definitely on bridge01 ... there's only one version of ansible installed ... https://zuul.opendev.org/t/openstack/build/08b5591ed79b4edcb1a68759c4fad5ef/console22:32
clarkbcatching up. yubioath-desktop stopped working for generating totp keys after updates... ykman oath account code foo still works so it isn't completely broken but I tried the backup device and different usb controllers and different computers... ugh22:36
ianwhttps://github.com/ansible/ansible/blob/devel/lib/ansible/modules/apt.py#L726 is what we hit22:38
clarkbI am able to u2f with the devices too. Must be an application layer bug (now to stop distracting people with key issues)22:39
ianwhaha distractions welcome22:39
ianwthis is from the last good run of base -> https://paste.opendev.org/show/bvaYbHtjeKs4cnRa8xI3/22:45
ianwit seems like the package: install is reporting changed on these hosts.  maybe these are our only focal hosts22:46
clarkbianw: I think others are focal too like all of zuul22:46
ianwyeah ...22:47
ianw# less base.yaml.log.2022-12-06T15:37:55 | grep elpa-yaml-mode | wc -l22:47
ianw522:47
*** rlandy|rover is now known as rlandy|out22:47
clarkbthe afs hosts were upgraded in place iirc maybe related to that?22:47
ianwahhh --- in place upgrades could very definitely be the difference with these 522:47
ianwdid we do that for lists.opensatck.org?22:47
clarkbianw: yes lists was inpace too in order to preserve the ip reputation22:48
clarkbwe're giving that up with lists01.opendev.org because hte inplace has run out of steam due to kernel problems22:48
ianwok ... well i think we have a partial smoking gun then 22:49
ianwhttps://paste.opendev.org/show/b3EAvluXhg5tOrHNNMHD/22:50
ianwit's really like it's running with a old/different version of ansible22:59
clarkbdid it maybe cache something on the client side to execute the installation?22:59
clarkbansible does copy that content over and maybe that is stale somehow?22:59
ianwthis is the bootstrap job from the same buildset, where we print the ansible ver -> https://zuul.opendev.org/t/openstack/build/f154fd86f9384c8cb273c7dc8b361ea9/console#2/1/16/bridge01.opendev.org22:59
ianwthere's no files in .ansible on afs0223:03
ianwhrm, i did get it just then with a standalone run -- i put "-f 50" on it like the main run does ...23:07
ianwnope, that didn't make afs01 hit it23:10
ianwalright, i don't know why, but afs01 doesn't hit it on manual testing, but afs02 does23:12
ianwhttps://paste.opendev.org/show/briE6350LK2YxBAKMFxj/ is the difference23:14
ianwwe could just purge and re-install this package and forget about it.  but it does annoy me that there must be an ansible issue here ...23:14
clarkbI'm ok with easy workarounds for things like this23:15
clarkb:)23:15
ianw# apt-cache show yaml-mode23:17
ianwN: Can't select versions from package 'yaml-mode' as it is purely virtual23:17
ianwon afs0123:17
ianw$ apt-cache show yaml-mode23:17
ianwPackage: yaml-mode23:17
ianwon afs0223:17
ianwHOW ARE THEY DIFFERENT!?!?!23:17
ianwrc  yaml-mode      0.0.9-2      all          emacs mode for YAML files23:22
ianw"removed but it's configuration files are still there"23:22
JayFapt-get purge {package} # gets you to uninstalled23:28
JayFbut will remove all the config/data dirs too, so only do that if you're sure that's OK23:28
ianwwhat's weird is that this doesn't have any conf files23:31
ianw$ dpkg -S yaml-mode | grep '^yaml-mode'23:32
ianwyaml-mode: /etc/emacs/site-start.d/50yaml-mode.el23:32
JayFThat doesn't count as one?23:32
ianwso we've removed the package, but it hasn't removed this file ... which is part of the package23:32
ianwhttps://packages.ubuntu.com/bionic/all/yaml-mode/filelist23:32
JayFyeah, purge will get rid of that23:32
JayFbecause it's in etc23:32
ianwwell apt should have got rid of it, it's one of the package files23:33
* JayF is fuzzy on modern ubuntu packaging but that's what he remembers from decades-old debian23:33
ianwi wonder if this leads us back to some sort of apt bug23:33
ianwwhen we've done the in-place dist-upgrade, it's done something like remove epel-mode and installed elpa-epel-mode.  but it hasn't actually removed one of the epel-mode files.  so now it appears as a zombie in dpkg, which is messing up ansible's apt-cache parsing23:35
ianwi think we went trusty->bionic->focal on these23:39
ianwok, in a trusty container, if i install then remove yaml-mode, it leaves behind23:42
ianw# dpkg -S yaml-mode23:42
ianwyaml-mode: /etc/emacs/site-start.d/50yaml-mode.el23:42
opendevreviewClark Boylan proposed openstack/project-config master: Fix nodepool API ops Grafana graphs  https://review.opendev.org/c/openstack/project-config/+/86705423:44
clarkbI love the screenshots cc ianw found some bugs23:44
ianwclarkb: :) there's instructions there to setup a container which makes fro pretty quick iteration too23:44
ianwhttps://docs.opendev.org/opendev/system-config/latest/grafana.html#local-development23:45
clarkbya in this case I juts have to ifddle resource paths23:46
ianwi could chase this all day.  i'm 99.99% certain it's to do with yaml-mode installing it's files into /etc and then them hanging around a conffiles23:56
ianwbut it's rather pointless ...23:56
ianwi'm just going to log into the 5 hosts and purge the package23:56

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!