13:00:04 <mnasiadka> #startmeeting kolla
13:00:04 <opendevmeet> Meeting started Wed Aug 30 13:00:04 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:04 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:04 <opendevmeet> The meeting name has been set to 'kolla'
13:00:07 <mnasiadka> #topic rollcall
13:00:08 <mnasiadka> o/
13:00:11 <mmalchuk> o/
13:00:53 <opendevreview> Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: Remove incorrect variable from OpenSearch role  https://review.opendev.org/c/openstack/kolla-ansible/+/893175
13:01:00 <mhiner> o/
13:01:07 <SvenKieske> o/
13:01:51 <bbezak> o/
13:03:11 <SvenKieske> it might be the case that frickler is busy with some router hardware, just pinged him privately
13:03:20 <jsuazows> o/
13:03:32 <mnasiadka> #topic agenda
13:03:32 <mnasiadka> * CI status
13:03:32 <mnasiadka> * Release tasks
13:03:32 <mnasiadka> * Regular stable releases (first meeting in a month)
13:03:32 <mnasiadka> * Current cycle planning
13:03:34 <mnasiadka> * Additional agenda (from whiteboard)
13:03:34 <mnasiadka> * Open discussion
13:03:39 <mnasiadka> #topic CI status
13:03:46 <mnasiadka> overall green I think, especially master
13:03:59 <mnasiadka> if there's anything broken - please update the whiteboard
13:04:04 <mnasiadka> #topic Release tasks
13:04:27 <mnasiadka> it's cycle highlights week
13:04:43 <mnasiadka> I'll try to compose something and will add let's encrypt and podman, because we should get it in
13:05:32 <SvenKieske> the LE change author seems pretty unresponsive since last september, see my message above. maybe someone could adopt the change? I guess it's only minor stuff needed
13:05:40 <kevko> i will work on comments ..
13:05:43 <mnasiadka> SvenKieske: kevko is now updating it
13:05:47 <SvenKieske> nice :)
13:05:51 <mnasiadka> ok, it's R-5 and we have one task
13:05:53 <mnasiadka> [all] Check for new versions of infrastructure components
13:05:59 <mnasiadka> #link https://docs.openstack.org/kolla/latest/contributor/release-management.html#r-5-cycle-highlights-deadline
13:06:04 <mnasiadka> anybody happy to pick this up?
13:06:23 <kevko> SvenKieske: did you diff between last patchsetset from jason and my last patchset ? :D
13:06:31 <SvenKieske> ok, time for dumb questions: what needs to be done? :D
13:06:39 <kevko> SvenKieske: there is massive change which is tested
13:06:55 <kevko> before it was not working properly
13:06:56 <SvenKieske> ah the link explains it
13:07:06 <mnasiadka> SvenKieske: there's a list of components listed, basically skim kolla docker images for versions and check if there are newer
13:07:29 <mnasiadka> I mean scan
13:08:25 <SvenKieske> I feel like that list is incomplete, no? if I should just check upstream I can. if this needs to be checked with supported distros etc. it's much more work.
13:08:46 <SvenKieske> is this just about upstream checking (reporting to..where?) I'm okay doing that.
13:09:24 <SvenKieske> help appreciated, as I notice this also includes all prometheus exporters :D
13:09:32 <mnasiadka> yeah, we should update that list
13:09:52 <mnasiadka> I think let's check only for versions that we control - as in prometheus exporters and similar things
13:10:27 <SvenKieske> so everything where we pull directly from upstream (repos)? still not clear to me, sorry.
13:10:41 <mnasiadka> yes
13:10:55 <SvenKieske> okay, understood, seems reasonable. :)
13:10:55 <mnasiadka> so in some cases we take a repo with some major version - like in rmq/mariadb case
13:11:06 <mnasiadka> I think we bumped RMQ and MariaDB already, so it's off the list
13:11:29 <mnasiadka> ceph is rather dependent on what we get from debian/ubuntu/centos repos
13:11:53 <mnasiadka> ok, if you need any help - shout out :)
13:11:58 <SvenKieske> cool
13:12:04 <mnasiadka> #topic Current cycle planning
13:12:15 <mnasiadka> So, Let's Encrypt - kevko will update based on comments
13:12:34 <mnasiadka> Podman - mhiner need any help, or we just need to start reviewing?
13:13:10 <mhiner> Yes, I need help. Can you look into whiteboard for tldr?
13:13:58 <mhiner> What I don't understand is why this issue only happens on aarch64 debian test case
13:14:17 <mnasiadka> K-A Podman patch https://review.opendev.org/c/openstack/kolla-ansible/+/799229 failed test: https://zuul.opendev.org/t/openstack/build/781e142dcb194767a130ed3670226aa5/logs
13:14:24 <mnasiadka> Seems like the ansible insde of kolla_toolbox is unable to write in it's log file just for this single test case. Why could that be?
13:14:35 <mnasiadka> kolla-ansible/ansible/roles/common/handlers/main.yml:2 fails with: [WARNING]: log file at /var/log/kolla/ansible.log is not writeable and we cannot create it, aborting
13:14:44 <mnasiadka> aarch64 only? huh
13:15:15 <SvenKieske> are we sure that the kolla_toolbox container has cap_audit? because that's a common way to not be able to write. I know the patchset enables this cap, but maybe not globally enough? just a random thought.
13:15:19 <opendevreview> Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration  https://review.opendev.org/c/openstack/kolla-ansible/+/893172
13:15:37 <mnasiadka> mhiner: https://63942ee63c42f3e33db1-2a052569d3facbabacb331e67cb0b17e.ssl.cf2.rackcdn.com/799229/84/check-arm64/kolla-ansible-debian-aarch64-podman/781e142/primary/logs/container_logs/kolla_toolbox.txt - that looks a bit weird?
13:15:51 <kevko> zun also failing for debian
13:16:00 <SvenKieske> mhiner: can you maybe add a simple "echo 'test' >> /var/log/kolla/ansible.log" somewhere to rule out bugs in ansible/python?
13:17:10 <SvenKieske> and maybe also an echo "test" > /tmp/testfile.txt or something to rule out permission problems with /var/log/
13:17:46 <mhiner> to be clear, inside of the kolla_toolbox right?
13:17:51 <SvenKieske> unless someone has a better idea to narrow it down. I didn't take a closer look.
13:18:00 <mhiner> and iirc, kolla_toolbox is in privileged mode so it should not need any additional caps
13:18:02 <SvenKieske> mhiner: sure, inside the container, with the same user
13:18:55 <SvenKieske> mhiner: right, if it's privileged it should suffice. but weird that it's only failing on aarch64 like mnasiadka mentioned.
13:19:39 <mnasiadka> well, basically as you can see we have no clue, but keep debugging :)
13:20:27 <mnasiadka> #topic Additional agenda (from whiteboard)
13:20:32 <mnasiadka> One was the podman one we discussed
13:20:36 <SvenKieske> we could do one recheck to rule out spurious sunflare errors in the DC :)
13:20:48 <mmalchuk> have opend discussion related to podman
13:21:24 <mnasiadka> ok, frickler - octavia jobboard - I think it's getting reviews
13:21:32 <mnasiadka> SvenKieske: asking for review/feedback on https://review.opendev.org/c/openstack/kolla-ansible/+/874769
13:22:00 <mnasiadka> I think we did speak about it last time, or outside of a meeting
13:22:03 <mmalchuk> commented octavia jobboard
13:22:16 <frickler> ack, thx mmalchuk
13:22:25 <SvenKieske> I
13:22:28 <mnasiadka> if the author would add some CI tests and move the python inline code to a normal file - would be nice
13:22:36 <SvenKieske> I'll try to poke the author of said patch
13:22:51 <SvenKieske> I forgot to remove it from the whiteboard
13:23:20 <mnasiadka> SvenKieske: the kolla patch you wanted to talk about is already merged - so I guess that concludes :)
13:23:49 <mnasiadka> k.yarovoy: podman for kolla - https://review.opendev.org/c/openstack/kolla/+/825791/10 one patchset left, but it will need some efforts on rebase and adaptation of ci. Is there any planned way to approach it?
13:23:50 <SvenKieske> yeah, didn't update the whiteboard, just did it, sorry.
13:24:04 <mnasiadka> ok then, would be nice to have support for podman build in Kolla
13:24:12 <kevko> btw, all container logs are broken https://63942ee63c42f3e33db1-2a052569d3facbabacb331e67cb0b17e.ssl.cf2.rackcdn.com/799229/84/check-arm64/kolla-ansible-debian-aarch64-podman/781e142/primary/logs/container_logs/index.html
13:24:18 <mnasiadka> mhiner: k.yarovoy is your colleague?
13:24:21 <opendevreview> Matt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/893137
13:24:28 <mhiner> mnasiadka: yes
13:24:49 <mnasiadka> mhiner: if he could rebase the patch (now it's in merge conflict) - I can help with CI
13:25:16 <opendevreview> Matt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/893137
13:25:28 <mnasiadka> mhiner: he's not here, so maybe you could convey the message :)
13:25:28 <mhiner> also, he did not write it bet he wanted to ask if someone could pick it up bcs he won't be able to attend it for at least two next weeks
13:25:36 <mnasiadka> ah ok
13:25:43 <mnasiadka> I'll try rebasing on my own then and adding CI
13:25:50 <mhiner> thank you
13:26:08 <SvenKieske> kevko: can you open a bug? I'm not sure what changed recently in the logging infrastructure
13:26:20 <SvenKieske> sample error msg: Error: invalid argument "all" for "--tail" flag: strconv.ParseInt: parsing "all": invalid syntax
13:26:23 <mnasiadka> SvenKieske: it's the podman patch - probably logging is broken only there
13:26:34 <mnasiadka> next one:
13:26:37 <mnasiadka> jsuazows - Easy bug fixes proposed:
13:26:37 <mnasiadka> OVS_DPDK task name fix
13:26:37 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/892613
13:26:37 <mnasiadka> Horizon SSO fix
13:26:37 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/891923
13:27:07 <SvenKieske> the horizon patch just needs people with these weird +2 powers ;)
13:27:35 <mnasiadka> yeah, let's hope it doesn't break anything :)
13:27:37 <mmalchuk> mnasiadka another one trivial: https://review.opendev.org/c/openstack/kolla-ansible/+/892837
13:27:41 <SvenKieske> I'll look at dpdk even if I have no real interest in it.
13:27:48 <mnasiadka> but I guess I did override that somewhere locally for SSO to work
13:28:04 <mnasiadka> dpdk is trivial fix
13:28:25 <SvenKieske> the SSO stuff was rather good tested, afaik. don't know if the github issue is linked somewhere
13:28:54 <mnasiadka> bbezak: willing to have a look in the SSO patch? ^^
13:30:16 <kevko> mnasiadka: dpdk approved
13:30:26 <mmalchuk> and server-status and kayobe patches from the last meeting question?)
13:31:10 <SvenKieske> nvm, I thought about another SSO bug, rather that one was about OIDC..
13:31:43 <mnasiadka> ok, those are done
13:31:53 <mnasiadka> mmalchuk: can you post a link so other cores can have a look?
13:32:14 <mmalchuk> #link https://review.opendev.org/c/openstack/kayobe/+/861397
13:32:20 <mmalchuk> #link https://review.opendev.org/c/openstack/kayobe/+/879554
13:32:36 <opendevreview> Jan Gutter proposed openstack/kolla-ansible master: etcd: update to v3.4  https://review.opendev.org/c/openstack/kolla-ansible/+/890465
13:32:59 <mmalchuk> #link https://review.opendev.org/c/openstack/kolla/+/891098
13:34:09 <mnasiadka> ok then
13:34:10 <kevko> https://review.opendev.org/q/owner:lukas.mrtvy%2540gmail.com << another sso fixes abandoned ... guesswhat8 abandoned
13:34:12 <mnasiadka> #topic Open discussion
13:34:34 <mmalchuk> 1. "bad practice"
13:34:34 <mmalchuk> https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892990
13:34:55 <mmalchuk> unset or comment?
13:37:07 <mnasiadka> I have no preference, but it seems like a cosmetic change with no gain
13:37:37 <mmalchuk> imho default+omit is a good practice
13:37:43 <mnasiadka> as long as we don't have coding guidelines with best practices documented and align all code to this - I don't think it makes any sense
13:38:10 <frickler> gain of good style, I'm +2 and we can add that to coding guidelines?
13:38:42 <SvenKieske> {{ virtualenv is none | ternary(omit, virtualenv) }}" is way harder to read than: {{ virtualenv | default(omit) }}
13:38:42 <mnasiadka> do we have any coding guidelines? :)
13:39:11 <frickler> didn't we link to that last week?
13:39:11 <mnasiadka> well, ternary is not there today, it's only afaik in podman patch
13:39:12 <SvenKieske> we could invent some :) I'm all for it, because some reviews boil down to taste. I think a guideline can help cut discussions short
13:39:50 <SvenKieske> it's the next best thing to: always do what the linter tells you
13:39:50 <mmalchuk> anyone have a link to coding guidelines?
13:40:16 <mnasiadka> first of all, that's not the only occurence of using "is not none" for an empty defined var in ansible-collection-kolla
13:40:21 <mnasiadka> so either we change all of them or none
13:40:23 <frickler> https://docs.openstack.org/kolla/latest/contributor/kolla-design-philosophy.html , maybe need to extend the domain from design to coding?
13:40:31 <SvenKieske> like the python "black" approach, or gofmt, or rustfmt
13:40:51 <mmalchuk> +1 for all of them)
13:40:54 <mnasiadka> second of all, maybe newer ansible-lint at least warns on something like that, then it would be easier to enforce
13:40:59 <mnasiadka> or we need to write a custom rule
13:41:19 <SvenKieske> so let me rephrase: I don't like coding guidelines, I like syntax enforced by a linter (basically coding guidelines automated as code). If I can't have that; I like guidelines.
13:41:33 <mmalchuk> ansible-lint quiet about this
13:41:43 <mnasiadka> we use some old version
13:41:47 <mnasiadka> maybe it's time to bump up
13:41:48 <mmalchuk> may be never version?
13:41:54 <mnasiadka> and then add some custom rules if needed
13:42:06 <SvenKieske> ansible-lint is really a pain because they, at least sometimes, change quite a lot of stuff which results in new errors and warnings
13:42:21 <frickler> maybe that cannot be caught by a linter, because it needs too much semantic understanding
13:42:35 <SvenKieske> mnasiadka: put ansible-lint on the "to check for updates" list? ;)
13:42:46 <mnasiadka> SvenKieske: might be, but that will be painful
13:43:35 <mmalchuk> so lets merge https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892990 and fix all other places later with lint
13:43:38 <mnasiadka> ok, anyway - I think linter that enforces it is the way
13:43:47 <mnasiadka> mmalchuk: I disagree
13:43:52 <mmalchuk> why?
13:44:34 <mmalchuk> starting from coding guidelines and lint ?
13:45:01 <mnasiadka> I agree with SvenKieske that writing coding guidelines is a waste of resources, it's better to do proper linting.
13:45:28 <mnasiadka> I'm fine with fixing bad practices, but we need to check for them and fix them everywhere, not in just one role.
13:45:29 <mmalchuk> upgrade lint and make job non voting?
13:45:49 <mnasiadka> upgrade lint and stack lint update on top of patches that make us compliant
13:46:01 <SvenKieske> I don't know if a tool can enforce good style on playbooks, the ansible-lint stuff is okayish, but had some drawbacks in the past. as always: a linter or coding guidelines will incur additional work
13:46:30 <SvenKieske> but it hopefully will make stuff more consistent and maybe also catch some errors
13:46:37 <mmalchuk> do we really have linter like black ?
13:46:54 <mmalchuk> which changes the code
13:47:33 <SvenKieske> ansible-lint --write .
13:47:51 <SvenKieske> I don't know how reliable that is, but it exists :)
13:48:24 <mmalchuk> wow... thanks. didn't know
13:48:39 <mmalchuk> will check all the code)))
13:50:47 <mnasiadka> ok then
13:50:48 <SvenKieske> as long as we realize that everything in live has also drawbacks it might be a worthwhile extension :)
13:51:02 <mnasiadka> anything else?
13:51:26 <jangutter> fans of ARA logging | https://review.opendev.org/c/openstack/kolla-ansible/+/891097  :-p
13:51:43 <mmalchuk> lack of kayobe cores
13:52:08 <mnasiadka> jangutter: I'm still not convinced around the additional time of jobs
13:52:10 <SvenKieske> you can scratch kayobe from that sentence ;)
13:52:21 <mmalchuk> :)
13:52:43 <mmalchuk> anyway
13:52:56 <SvenKieske> regarding ARA: It might be nice to be able to enable it on a case-by-case basis?
13:53:13 <opendevreview> Michal Arbet proposed openstack/kolla-ansible master: Add support of podman deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/799229
13:53:14 <opendevreview> Michal Arbet proposed openstack/kolla-ansible master: Fix podman logs  https://review.opendev.org/c/openstack/kolla-ansible/+/893187
13:53:36 <jangutter> mnasiadka : that is indeed a fair reason not to do it. I can easily change it so that it can be triggered by something in the commit message.
13:54:06 <tony> Hello Team, .. I am facing the issue regarding installing new compute nodes on my wallaby openstack cluster using kolla-ansible, I am getting the error below
13:54:08 <mnasiadka> we might go that path, ara might be sometimes useful
13:54:16 <mnasiadka> tony: wait a moment, there's a meeting in progress
13:54:34 <mnasiadka> jangutter: can you rework that please?
13:54:40 <tony> Oooh My Bad
13:54:52 <mmalchuk> btw hell of ansible-lint in bifrost: https://review.opendev.org/q/project:openstack/bifrost+topic:ansible-lint
13:55:20 <jangutter> mnasiadka: ack, won't take too long - we happy with the interface where "#log_ara" or something in the commit message triggers it?
13:55:31 <mnasiadka> jangutter: I think so
13:55:50 <mnasiadka> jangutter: we could add docs around how to use it somewhere in Kolla-Ansible contributor guide?
13:56:11 <jangutter> mnasiadka: good idea, I'll add it there!
13:56:14 <mnasiadka> thanks!
13:56:22 <mnasiadka> ok, I guess it's enough for today :)
13:56:25 <mnasiadka> Thanks all for coming!
13:56:27 <mnasiadka> #endmeeting