| SvenKieske | mnasiadka, kevko: for opensearch I tried this but it failed then with grafana: https://review.opendev.org/c/openstack/kolla/+/915322 should be good to merge imho | 06:51 |
|---|---|---|
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: Run kayobe-tox-ansible using Ubuntu Focal https://review.opendev.org/c/openstack/kayobe/+/915330 | 07:23 |
| SvenKieske | kevko, mnasiadka: I raised also a bug yesterday, but "only" to pin opensearch https://bugs.launchpad.net/kolla/+bug/2060668 | 07:52 |
| SvenKieske | and what do we do for the grafana situation? it seems grafana has it's own conference ongoing so it seems unlikely someone will respond soon, given it's been broken for a week now | 07:54 |
| SvenKieske | does someone know anyone at grafana labs per chance? | 07:54 |
| SvenKieske | else I would just say mark the whole monitoring stuff as unbuildable and non-voting for now? I know that's a rather harsh reaction, but CI is blocked. | 07:55 |
| SvenKieske | mhm, there are reports coming in that it works again, will test locally | 07:57 |
| SvenKieske | \o/ seems to work again (tested in debian latest container) | 08:01 |
| SvenKieske | fwiw there was an incident but only related to the grafana-agent apt package: https://status.grafana.com/incidents/xblqfhgkz0sg | 08:05 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Revert "Mark kayobe-tox-ansible job as non-voting" https://review.opendev.org/c/openstack/kayobe/+/915465 | 08:21 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [DNM] Add apt-cache policy to identify problems with install package https://review.opendev.org/c/openstack/kolla/+/915466 | 08:38 |
| SvenKieske | kevko: the grafana stuff seems to have just been silently fixed on their part, that's why your local installation works again. :) | 08:45 |
| kevko | kevko: I don't think | 08:51 |
| kevko | SvenKieske: what does it mean they just silently fixed ? I've tried to build grafana right after build failed in CI ..and worked ... so it can be problem in a mirror i would say ...as my local build refers directly to their repo while CI not | 08:52 |
| SvenKieske | kevko: I pinged you multiple time on issues raised and fixes proposed, did you read any of that? | 08:53 |
| SvenKieske | kevko: one last try: https://github.com/grafana/grafana/issues/85827 many people had this issue with direct upstream repo officially from grafana | 08:53 |
| SvenKieske | kevko: I also could reproduce this issue locally yesterday, with upstream repo, no mirrors involved | 08:54 |
| SvenKieske | kevko: if you read this issue till the end it started working again today, one or two hours ago | 08:54 |
| SvenKieske | kevko: I also verified this locally. so it really really was broken upstream. for some days at least. | 08:55 |
| SvenKieske | HTH :) | 08:55 |
| kevko | SvenKieske: haha, you are right :D ...working now | 09:03 |
| SvenKieske | kevko: yeah, good that it works now. I was actually also quite surprised that it was really an upstream issue | 09:04 |
| kevko | SvenKieske: hack with policy .... btw ... useful no ? https://8c8da465335a8767dcb7-4a7e6fa54cc64c7867564a09e7ad64c3.ssl.cf1.rackcdn.com/915466/1/check/kolla-build-debian/001517e/kolla/build/grafana.log | 09:04 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Fix handling configs in base image https://review.opendev.org/c/openstack/kolla/+/915440 | 09:06 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [DNM] Add apt-cache policy to identify problems with install package https://review.opendev.org/c/openstack/kolla/+/915466 | 09:06 |
| SvenKieske | mhm, maybe a little long? it would be nice if we could just toggle this via some debug switch, maybe? | 09:06 |
| kevko | SvenKieske: long ? :D ...we can also drop the log and write OK or FAIL into the log :D | 09:08 |
| SvenKieske | I mean it's not thaaat long, but not short? :D It's okayish I guess. | 09:16 |
| kevko | how is it possible that https://zuul.opendev.org/t/openstack/build/4e86289c85434261a24b6eca03716795 failing but locally working ? :D | 09:21 |
| kevko | ah, ok now failing :) | 09:27 |
| SvenKieske | so: do we want to keep opensearch non voting or do the pinning on 2.12 instead until we have auth working in CI? | 09:34 |
| kevko | how i hate unit tests :( | 09:45 |
| SvenKieske | I like them :) | 09:51 |
| priteau | SvenKieske: I would think we need pinning since it also needs to be backported? | 10:40 |
| opendevreview | Merged openstack/kayobe master: Revert "Mark kayobe-tox-ansible job as non-voting" https://review.opendev.org/c/openstack/kayobe/+/915465 | 11:12 |
| SvenKieske | priteau: my idea was to not extend the pinning outside master because it's a CI only issue and we would pin users to a possibly older release than what they are already running, also 2.13 fixed some cves, which I don't know the impact of. | 11:22 |
| SvenKieske | so afaik the idea was: workaround the CI issue for now by pinning and then implement proper auth between opensearch and haproxy so we can revert the pin asap | 11:23 |
| mnasiadka | Pinning opensearch is complicated in stable branches, we will break people already using 2.13 | 11:27 |
| mnasiadka | I’ll have a good look next week how to fix that drama | 11:27 |
| kevko | SvenKieske: to write them ? | 11:30 |
| SvenKieske | kevko: to write what? Sorry, I don't understand? | 11:35 |
| kevko | SvenKieske: how i hate unit tests -> I like them :) -> to write them ? | 11:36 |
| kevko | do anybody know ? https://6a83944802eed202968f-4cf4bb096d4774f141f32d30757edf40.ssl.cf2.rackcdn.com/915440/3/check/kolla-ansible-rocky9-upgrade/b6af6a4/primary/logs/kolla/libvirt/libvirtd.txt | 11:36 |
| SvenKieske | ah! :D well I like working unit tests that prevent regressions. ;) I can also write them, yes. | 11:36 |
| SvenKieske | yes there's a bug open about /dev/kvm | 11:37 |
| kevko | SvenKieske: I can also write them is not I like write them :D | 11:38 |
| SvenKieske | kevko: ah no, sorry, that one is slightly different: https://bugs.launchpad.net/kolla-ansible/+bug/2049716 | 11:38 |
| kevko | this is upgrade jobs for both debuntu and rocky | 11:39 |
| SvenKieske | kevko: we could probably run this before to check if everything works? https://libvirt.org/manpages/virt-host-validate.html | 11:39 |
| SvenKieske | I stole that idea from this issue :D https://github.com/kubevirt/kubevirt/issues/10605 | 11:39 |
| SvenKieske | might be an issue with nested-vt and/or our qemu-kvm driver config, or some distro package? but if it's affecting both rpm and deb it's either upstream bug or our config/env is buggy I think, without deep analysis. | 11:41 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.2: Revert "Mark kayobe-tox-ansible job as non-voting" https://review.opendev.org/c/openstack/kayobe/+/915355 | 11:47 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Revert "Mark kayobe-tox-ansible job as non-voting" https://review.opendev.org/c/openstack/kayobe/+/915356 | 11:47 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add container engine migration scenario https://review.opendev.org/c/openstack/kolla-ansible/+/836941 | 13:09 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Remove "distutils" library https://review.opendev.org/c/openstack/kolla-ansible/+/857003 | 13:32 |
| opendevreview | Alex Welsh proposed openstack/kolla-ansible master: Automate prometheus blackbox configuration https://review.opendev.org/c/openstack/kolla-ansible/+/912420 | 14:39 |
| opendevreview | Sven Kieske proposed openstack/kolla master: CI/Master only: pin opensearch{-dashboards} https://review.opendev.org/c/openstack/kolla/+/915322 | 15:28 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Add backend TLS between MariaDB and ProxySQL https://review.opendev.org/c/openstack/kolla-ansible/+/909912 | 15:31 |
| chris218 | Hi I've been wondering recently, does Kolla support virtualized control plane setup? As in control plane being inside VMs managed by itself? | 17:06 |
| frickler | chris218: well kolla uses docker or podman containers, so that's kind of virtualized. I don't see how you could run the control plane within VMs in nova, how would you bootstrap or cold start such a setup? | 17:44 |
| chris218 | Create 3 VMs manually with QEMU on the same network that Neutron will use, then deploy on all hosts + 3 VMs, and lastly manually shut down and import one-by-one each VM to Nova keeping all the static IPs | 17:47 |
| chris218 | That would be the rough idea | 17:47 |
| chris218 | I've inherited 7 pretty beefy servers to manage which have a year old and a bit broken Kolla installation and am now evaluating different ways I can setup it most efficiently | 17:52 |
| chris218 | Hmm the only issue might be if I stop all the containers and then want to start them I won't be able to connect to the control plane to start services on it back 🤔 | 21:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!