Friday, 2023-12-01

opendevreviewMerged openstack/kayobe master: switches: add flag to save configuration on Dell switches  https://review.opendev.org/c/openstack/kayobe/+/86261302:22
opendevreviewMerged openstack/kayobe master: Install docker Python package in kolla venv  https://review.opendev.org/c/openstack/kayobe/+/90228204:08
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: fluentd: Fix getting podman labels  https://review.opendev.org/c/openstack/kolla-ansible/+/90235206:52
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: magnum: support kubeconfig configuration file  https://review.opendev.org/c/openstack/kolla-ansible/+/89629207:15
mnasiadkaok then, last two patches for k-a ^^07:16
SvenKieskeo/08:07
SvenKieskelast two patches..so far! /homer_meme08:07
SvenKieskemnasiadka: the kubeconfig check in magnum should be optional/allowed to fail, no? :)08:10
mnasiadkaSvenKieske: if the file isn't there, we don't template it in config.json08:31
mnasiadkaand don't copy it08:31
mnasiadkaso it should be fine08:31
SvenKieskeok08:32
mnasiadkajust replied in gerrit08:32
mnasiadkaok, kolla seems to be ready, let me update the release patch08:38
mnasiadkaand kolla-ansible in a few hours after those two will merge08:40
SvenKieskenice :)08:43
priteauAnyone able to approve this trivial backport? https://review.opendev.org/c/openstack/kolla-ansible/+/90171508:59
opendevreviewPierre Riteau proposed openstack/kolla-ansible master: Support CloudKitty deployment with internal TLS  https://review.opendev.org/c/openstack/kolla-ansible/+/86659809:01
mnasiadkadone09:05
priteauthanks!09:14
opendevreviewSven Kieske proposed openstack/kolla-ansible master: Add ovn-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/85549809:19
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: CI: Fail on fluentd log parsing errors  https://review.opendev.org/c/openstack/kolla-ansible/+/90223909:21
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/zed: fluentd: Use labels for transition to v5  https://review.opendev.org/c/openstack/kolla-ansible/+/90224009:22
mnasiadkawrong patch :D09:23
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/zed: CI: Fail on fluentd log parsing errors  https://review.opendev.org/c/openstack/kolla-ansible/+/90238109:23
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/zed: CI: Fail on fluentd log parsing errors  https://review.opendev.org/c/openstack/kolla-ansible/+/90238109:25
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/yoga: CI: Fail on fluentd log parsing errors  https://review.opendev.org/c/openstack/kolla-ansible/+/90238209:25
* frickler is curious how many backports might be missing for that09:29
mnasiadkathat's what I want to check :)09:29
fricklermnasiadka: do you also want to propose rc2 for a-c-k or shall I?09:29
mnasiadkafrickler: you can do that, I hope it's going to be the only rc209:29
fricklerack09:30
fricklerhttps://review.opendev.org/c/openstack/releases/+/90236409:34
opendevreviewMerged openstack/kolla-ansible master: fluentd: Fix getting podman labels  https://review.opendev.org/c/openstack/kolla-ansible/+/90235210:09
mnasiadkaok, one more and time to update release patch10:14
opendevreviewMerged openstack/kolla-ansible stable/2023.1: Move [oslo_policy] back inside Jinja if block  https://review.opendev.org/c/openstack/kolla-ansible/+/90171510:24
opendevreviewMerged openstack/kayobe stable/zed: CI: Test SELinux configuration  https://review.opendev.org/c/openstack/kayobe/+/90123911:18
opendevreviewWill Szumski proposed openstack/kayobe master: Stop kolla fighting with kayobe over selinux state  https://review.opendev.org/c/openstack/kayobe/+/90236911:18
opendevreviewMerged openstack/kolla-ansible master: magnum: support kubeconfig configuration file  https://review.opendev.org/c/openstack/kolla-ansible/+/89629211:22
mnasiadkayay11:26
opendevreviewWill Szumski proposed openstack/kayobe master: Disable configuration of SELinux by Kolla Ansible  https://review.opendev.org/c/openstack/kayobe/+/90238511:33
opendevreviewOpenStack Release Bot proposed openstack/kolla stable/2023.2: Update .gitreview for stable/2023.2  https://review.opendev.org/c/openstack/kolla/+/90237511:55
opendevreviewOpenStack Release Bot proposed openstack/kolla stable/2023.2: Update TOX_CONSTRAINTS_FILE for stable/2023.2  https://review.opendev.org/c/openstack/kolla/+/90237611:55
opendevreviewOpenStack Release Bot proposed openstack/kolla master: Update master for stable/2023.2  https://review.opendev.org/c/openstack/kolla/+/90237711:55
opendevreviewWill Szumski proposed openstack/kayobe master: Remove unused kolla bootstrap variables  https://review.opendev.org/c/openstack/kayobe/+/90237912:00
opendevreviewPierre Riteau proposed openstack/kayobe master: Remove kolla_enable_host_ntp variable  https://review.opendev.org/c/openstack/kayobe/+/90238012:02
opendevreviewWill Szumski proposed openstack/kayobe master: Remove unused kolla bootstrap variables  https://review.opendev.org/c/openstack/kayobe/+/90237912:07
opendevreviewMerged openstack/kayobe stable/2023.1: CI: Test SELinux configuration  https://review.opendev.org/c/openstack/kayobe/+/90124312:14
opendevreviewOpenStack Release Bot proposed openstack/kolla-ansible stable/2023.2: Update .gitreview for stable/2023.2  https://review.opendev.org/c/openstack/kolla-ansible/+/90240212:14
opendevreviewOpenStack Release Bot proposed openstack/kolla-ansible stable/2023.2: Update TOX_CONSTRAINTS_FILE for stable/2023.2  https://review.opendev.org/c/openstack/kolla-ansible/+/90240312:14
opendevreviewOpenStack Release Bot proposed openstack/kolla-ansible master: Update master for stable/2023.2  https://review.opendev.org/c/openstack/kolla-ansible/+/90240412:14
frickleryay \o/12:17
opendevreviewMerged openstack/kolla stable/2023.2: Update .gitreview for stable/2023.2  https://review.opendev.org/c/openstack/kolla/+/90237512:28
opendevreviewMerged openstack/kolla stable/2023.2: Update TOX_CONSTRAINTS_FILE for stable/2023.2  https://review.opendev.org/c/openstack/kolla/+/90237612:28
opendevreviewMerged openstack/kolla master: Update master for stable/2023.2  https://review.opendev.org/c/openstack/kolla/+/90237712:28
opendevreviewMerged openstack/kolla-ansible master: Update master for stable/2023.2  https://review.opendev.org/c/openstack/kolla-ansible/+/90240412:37
opendevreviewMerged openstack/kolla-ansible stable/2023.2: Update .gitreview for stable/2023.2  https://review.opendev.org/c/openstack/kolla-ansible/+/90240212:39
opendevreviewMerged openstack/kolla-ansible stable/2023.2: Update TOX_CONSTRAINTS_FILE for stable/2023.2  https://review.opendev.org/c/openstack/kolla-ansible/+/90240312:40
SvenKieske🥳12:58
opendevreviewMerged openstack/kayobe master: Remove libgcrypt package update  https://review.opendev.org/c/openstack/kayobe/+/90182512:58
dcapone2004i do not know what it is, but  this version of KA and I are not getting along...13:02
dcapone2004for some version my ceph.conf file is not being copied into the nova_compute container and that is preventing nova_compute from working properly13:03
dcapone2004did the config directory change for this?  i have ceph.conf and the keyring file along with the nova-compute.conf in /etc/kolla/nova13:05
dcapone2004cinder and glance work fine and the ceoh.conf and keyrings copied as expected13:08
dcapone2004sorry i have the files in /etc/kolla/config/nova...typed too fast13:08
ironfootCould I get someone to give these a final look and +W  please? https://review.opendev.org/q/Iddbdc4190b7953e9140d0740daf57f4062ba1b7613:11
sylvrHello! I would like to test a git "cherry pick" to see if it fixed the issue I encountered, unfortunately I'm not sure how to proceed.. https://review.opendev.org/c/openstack/bifrost/+/86478713:13
ironfootsylvr: do you want to cherry-pick that change locally in your env?13:19
sylvryes!13:19
ironfootit's so small, it might be easier to just change the file manually13:19
sylvrI installed kayobe using source, so I have the git repo13:19
ironfootah, thanks for the context, as I haven't used kayobe before13:20
ironfootin the top-right there's a 3-dots menu13:21
ironfootthere's the option "download patch"13:21
ironfootyou can try that13:21
ironfootthere's a cherry-pick option13:21
sylvroh yeah thanks I don't know how I managed to miss that13:21
ironfootsylvr: TIL that menu exist, thanks for making me look 13:22
sylvrironfoot : do you know where the source of bifrost is supposed to be when using kayobe ? ^^13:24
ironfootI don't know, sorry13:24
sylvrokay, thanks ! I'll look around13:25
ironfootnp! :)13:25
mnasiadkafrickler: I’ll run some series of CI runs to check for obvious errors - if everything will be fine we’re good to aim for final before end of next week13:34
opendevreviewSimon Dodsley proposed openstack/kolla-ansible master: Add Pure Storage FlashBlade as Manila backend  https://review.opendev.org/c/openstack/kolla-ansible/+/87984614:26
opendevreviewSimon Dodsley proposed openstack/kolla-ansible master: Update Pure Storage NVMe Cinder driver  https://review.opendev.org/c/openstack/kolla-ansible/+/87984414:28
simondodsleycan i get some core reviews for https://review.opendev.org/c/openstack/kolla-ansible/+/879846 and https://review.opendev.org/c/openstack/kolla-ansible/+/879844. Just had to rebase as they have been hanging around so long.14:29
jovialsylvr: You'll need to rebuild the bifrost container to apply that bifrost patch (or patch the file in the container)14:31
mnasiadkasimondodsley: we just released Bobcat, give us couple of weeks to stabilise that, and then we can review C cycle patches14:34
simondodsleymnasiadka: Thanks. i did submit these in July for Bobcat as one in particular pertained to a new Cinder driver in Bobcat. I guess Caracal will be OK14:36
simondodsleyreally i'd like to get at least the NVMe driver backported to Bobcat14:37
mnasiadkaWe can discuss that once they are merged in master - but usually we don't backport features14:37
opendevreviewPierre Riteau proposed openstack/kayobe master: [release] Sync with Kolla Ansible for 2023.2  https://review.opendev.org/c/openstack/kayobe/+/90242114:54
SvenKieskemnasiadka: this fix for rmq in yoga though would be nice, especially since we introduced quorum queues now: https://review.opendev.org/c/openstack/kolla-ansible/+/90191914:57
mnasiadkaSvenKieske: done14:59
opendevreviewPierre Riteau proposed openstack/kayobe master: Use OpenStack 2023.2 release  https://review.opendev.org/c/openstack/kayobe/+/90242214:59
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: enable quorum queues  https://review.opendev.org/c/openstack/kolla-ansible/+/90238715:00
mnasiadkaactually, around quorum queues, let's try backporting that without changing the default15:00
opendevreviewPierre Riteau proposed openstack/kayobe-config master: [release] Synchronise with kayobe for 2023.2  https://review.opendev.org/c/openstack/kayobe-config/+/90242315:01
SvenKieskeokay; I'll do that15:01
SvenKieskeah there is already a cherry pick for 2023.115:02
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: enable quorum queues  https://review.opendev.org/c/openstack/kolla-ansible/+/90238715:02
mnasiadkayeah, just did that15:02
mnasiadkalet's see if that passes15:02
opendevreviewPierre Riteau proposed openstack/kayobe-config master: [release] Synchronise with kayobe for 2023.2  https://review.opendev.org/c/openstack/kayobe-config/+/90242315:03
mnasiadkaSvenKieske: wondering if we need a job that tests that, but that would need to be multinode for best coverage15:03
SvenKieskeyeah, maybe15:04
SvenKieskein theory we should also align the commit message to match what we do, because this does not enable quorum queues15:05
mnasiadkawe could just do a separate DNM patch that tests it does work15:07
mnasiadkawhich probably makes sense to do, just to make sure oslo.messaging is working properly at that branch level15:11
opendevreviewPierre Riteau proposed openstack/kayobe-config-dev master: [release] Synchronise with kayobe for 2023.2  https://review.opendev.org/c/openstack/kayobe-config-dev/+/90242515:12
SvenKieskesounds good15:13
mnasiadkabut it's Friday ;)15:14
SvenKieskeyes!15:16
opendevreviewPierre Riteau proposed openstack/kayobe master: Remove unused kolla bootstrap-servers variables  https://review.opendev.org/c/openstack/kayobe/+/90237915:24
opendevreviewMerged openstack/kolla-ansible stable/yoga: Fix kolla_container_engine since not exist in Yoga  https://review.opendev.org/c/openstack/kolla-ansible/+/90191916:16
dcapone2004can someone guide me to the correct playbook/file where external tls configuration is handled?  KA is seemingly completely ignoring the external tls certificate setting for some reason and generating a self signed16:18
mnasiadkak-a is only generating self signed if you run kolla-ansible generate-certificates16:20
mnasiadkaor something like that16:21
dcapone2004that is how it is supposed to work, but for some reason it isn't16:21
dcapone2004did a destroy16:21
dcapone2004and a deploy, even changed the filename on the comment external cert line to haxy.pem to see what it did, and I still ended up with a self signed haproxy.pem file16:22
dcapone2004that is being used to secure external tls16:22
dcapone2004https://pastebin.com/zSJBLVve16:24
dcapone2004that is my globals.yml configuration ... the path for the cert exists... I put it in the config subdirectory intentionally as when moving our customizations cluster to cluster it is easier to just move the entire config directory and have the cert there16:25
SvenKieskedcapone2004: I assume you have read https://docs.openstack.org/kolla-ansible/latest/admin/tls.html ?16:32
dcapone2004yes16:34
dcapone2004that is exactly how it is configured16:34
SvenKieskeyou did not set "kolla_copy_ca_into_containers", why?16:35
dcapone2004I'm ONLY using external tls in this case....as u can see in the globals.yml16:35
dcapone2004internal an dexternal ip and fqdn are different16:35
dcapone2004the default is no, and the certificate is a signed cert from a trusted authority16:35
SvenKieskeis the CA present in your containers?16:36
SvenKieskeare the permissons for the cert in "kolla_external_fqdn_cert" path correct?16:37
dcapone2004the cert isn't, but I would assume the root cert is16:37
dcapone2004in the doc u linked, that setting was for copying "the CA certificate files in /etc/kolla/certificates/ca will be copied into service containers to enable trust for those CA certificates. This is required for any certificates that are either self-signed or signed by a private CA, and are not already present in the service image trust store.16:38
dcapone2004so I didn't think it needed to be set to yes to copy just my cert file...16:38
dcapone2004but yes permissions are correct....no errors from KA16:38
SvenKieskewell it's for the cert chain/the CA, if the CA is in the container image already it should not be necessary, just double checking16:39
SvenKieskeare you building your own containers or reusing the public registry?16:40
dcapone2004I'm running a reconfigure now with the cert moved into the /etc/kolla/certificates path (updated globals.yml to this path as well)... testing if support for a path outside of that directory was lost somewhere / somehow16:40
dcapone2004public registry16:40
SvenKieskemhm, I'm confused, as I don't use the public images very infrequently, shouldn't there be more tags? https://quay.io/repository/openstack.kolla/ubuntu-source-haproxy?tab=tags16:49
dcapone2004I don't know if that was related to my issue ... but I am using rocky16:53
dcapone2004no change with cert moved16:56
SvenKieskemhm, yeah very weird16:56
dcapone2004enable copy_ca generates error because ca folder doesn't exist16:56
SvenKieskesrc or dst?16:57
dcapone2004pretty sure source16:57
dcapone2004/etc/kolla/certificates/ca doesn't exist16:57
dcapone2004"Could not find or access '/etc/kolla/certificates/ca/' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option16:58
SvenKieskewell if you have no special CA or the CA isn't to new for the container cert store this should all not be necessary so that is probably fine and should be set to "off" as you originally had it.17:03
SvenKieskeit doesn't explain the self signed certs though17:03
SvenKieskecan you maybe look at the creation timestamps of the self signed certs, are they really new or maybe some old artifacts of your deployment?17:04
dcapone2004interesting, as the cert does have a creation date and time that is before I did a KA destroy17:05
dcapone2004but I don't understand why KA wouldn't see the new cert and update anyway...  I have used KA and reconfigure to update certs at twice on previous KA versions without an issue17:06
dcapone2004at least*17:06
SvenKieskegood question :)17:07
dcapone2004lol17:07
dcapone2004does KA "cache" if the "certificates" command was run at any point somewhere?17:07
SvenKieskeis this a manual process or do you have maybe some CI stuff setup somewhere/some - new - automation which might interact with certificates?17:07
dcapone2004nope manual process, we are small :-)17:08
SvenKieskewell the "certificate" command generates the certs, I'm not 100% sure if the destroy command cleans that up? I never used that stuff manually afaik, or it is a long time ago, I can't remember.17:09
dcapone2004after i ran destroy17:09
dcapone2004I wiped out the entire /etc/kolla directory17:09
SvenKieskemaybe some path changed and the destroy command only cleans up some older path?17:09
SvenKieskemhm17:09
SvenKieskebut the cert is there with an old date? how did you do the wipe?17:09
dcapone2004I did copy the globals.yml and passwords.yml file out of the directory before deleting and copied it back, as I didn't want to have to reedit the file again manually17:10
SvenKieskesure17:10
dcapone2004rm -rf /etc/kolla17:10
dcapone2004I also tried cp our cert to /etc/kolla/haproxy.pem, so bbasically our is now in the haproxy.pem file, ran reconfigure, and it doens't look like KA noticed the change (understandable) as basically all tasks were skipped17:13
SvenKieskewhich version are you using again?17:13
dcapone20042023.117:13
dcapone2004the latest stable KA branch17:14
dcapone2004kolla-ansible@stable/2023.117:14
dcapone2004with haproxy overwritten, I am attempting to disable tls, run reconfigure, enable it, and run reconfigure again and see what happens17:15
SvenKieskeI'm going through the changelog, seeing if I find something related there..maybe we introduced a bug somehow17:16
SvenKieskedcapone2004: what changed since the last time you new this was working? did it work on stable/2023.1 for you before?17:28
dcapone2004this is the first time I am using 2023.117:32
dcapone2004we only use Openstack for dev, so we do not continuous update, basically deploy new when we upgrade hardware...although I was thinking of changing that as we have a lot of older hardware basically sitting idle now that I was going to play with17:33
SvenKieskemhm, but it did work on the release before that?17:33
dcapone2004last release we used was pypi release17:34
dcapone2004I think it is in the 15.X chain17:34
dcapone2004whatever the Yoga release was17:36
dcapone2004but this i feel would be a pretty common deployment concept, the last thing I envisioned not working right... and since it just ins't working for me or I am assuming you would have received more feedback already, I'm guessing it is a specific combo of 17:39
dcapone2004things causing the issue17:39
dcapone2004so tls is now off17:40
opendevreviewMerged openstack/kayobe master: Remove kolla_enable_host_ntp variable  https://review.opendev.org/c/openstack/kayobe/+/90238017:40
opendevreviewMerged openstack/kayobe-config-dev master: [release] Synchronise with kayobe for 2023.2  https://review.opendev.org/c/openstack/kayobe-config-dev/+/90242517:40
opendevreviewMerged openstack/kayobe-config master: [release] Synchronise with kayobe for 2023.2  https://review.opendev.org/c/openstack/kayobe-config/+/90242317:40
dcapone2004I am going to delete the /etc/kolla/haproxy/haproxy.pem file, enable tls, and run reconfigure17:41
SvenKieskeyes, I would have expected more reports when this was an issue on our side, as I'm fairly certain this is an often used configuration.17:41
dcapone2004haproxy.pem is deleted17:42
dcapone2004find /etc/kolla/* -name *.pem17:42
dcapone2004ONLY returns our signed cert17:42
dcapone2004I am going to run reconfigure now and see what happens and if it puts that haproxy.pem file back17:43
dcapone2004and of course /etc/kolla/haproxy/haproxy.pem is back as a self signed cert17:51
dcapone2004it is like the kolla_external_fqdn_cert variable is not being merged or picked up17:51
dcapone2004which was my first thought 30 hours ago, but I keep verifying that there is not a typo in the variable name17:53
SvenKieskeyou could use `inotifywait` on the parent directory to detect what creates the cert there: https://stackoverflow.com/a/7013328517:53
dcapone2004is that going to give me more info than openssl which is what I am guessing an ansible playbook is running?17:55
SvenKieskeno, well you could also just inspect the ansible.logs I think, that should tell you if a task ran that created the cert.17:57
dcapone2004I feel as those in prior KA release, it generated an error that haproxy.pem didn't exist (even if you tried to use the defaults and forgot to run the certificates command first)17:58
dcapone2004so I am wondering if there was a change implemented at some point that generates the cert if it is missing instead of erroring?17:59
SvenKieskeI looked at the stable/2023.1 changes only, I didn't find anything there. It might be good to write a detailed bugreport over at bugs.launchpad.net though18:01
SvenKieskeI'm off for the weekend now, bye! o/18:02
dcapone2004have a good weekend18:02
dcapone2004so adding to this, so it is logged in the irc logs...I updated the kolla_external_fqdn_cert variable to be an invalid path and KA errors right away, so it is seeing the variable and verifying that the file exists at the path specified18:26
dcapone2004but it seemingly seems to ignore it anyway, so the only thing I can think of is a merge issue18:26
dcapone2004omg....no bug...no issues....just working too fast and then ignoring the obvious leading to a wasted day18:46
opendevreviewPierre Riteau proposed openstack/kayobe master: Add python3-devel to Kayobe dependencies  https://review.opendev.org/c/openstack/kayobe/+/90244921:54
opendevreviewPierre Riteau proposed openstack/kayobe stable/yoga: Use latest published CentOS Stream 8 image  https://review.opendev.org/c/openstack/kayobe/+/90245122:30

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!