14:00:01 <mnasiadka> #startmeeting kolla
14:00:01 <opendevmeet> Meeting started Wed Nov 27 14:00:01 2024 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:01 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:01 <opendevmeet> The meeting name has been set to 'kolla'
14:00:04 <mnasiadka> #topic rollcall
14:00:05 <mnasiadka> o/
14:00:07 <SvenKieske> ah yeah, if you have created a zone in a different project you can have zone highjacks..
14:00:09 <SvenKieske> o/
14:00:10 <mmalchuk> o/
14:00:12 <kevko> \o/
14:00:43 <kevko> SvenKieske: but i thought that for this reason in neutron conf is designate user configured ...for this integration
14:01:14 <bbezak> \o
14:01:26 <SvenKieske> let's discuss this later :)
14:01:29 <mnasiadka> #topic agenda
14:01:30 <kevko> yep
14:01:35 <mnasiadka> * CI status
14:01:41 <mnasiadka> * Release tasks
14:01:51 <mnasiadka> * Current cycle planning
14:01:57 <mnasiadka> * Additional agenda (from whiteboard)
14:02:02 <mnasiadka> * Open discussion
14:02:04 <frickler> o/
14:02:06 <mnasiadka> #topic CI status
14:02:12 <mnasiadka> So... Docker doesn't like us
14:02:30 <mnasiadka> for Kolla - there are now only two images that we fetch from docker hub - debian and ubuntu base images
14:02:38 <SvenKieske> just mentioning this: https://www.docker.com/community/open-source/application/
14:03:00 <SvenKieske> other solutions are using quay.io or self hosting/building of course. not sure what's the least amount of work
14:03:03 <mnasiadka> I see two options of trying to fix this - 1) docker login 2) mirror those images to something like quay.io
14:03:11 <mnasiadka> No, I'm not going to beg Mirantis to give me access
14:03:25 <frickler> there is a role being developed in zuul to perform 2) https://review.opendev.org/c/zuul/zuul-jobs/+/935574
14:03:26 <mnasiadka> If somebody else wants to go the extra mile of paperwork - fine by me
14:04:30 <mnasiadka> Any volunteers for the paperwork? :)
14:05:05 <mnasiadka> Guess not
14:05:21 <mnasiadka> frickler: that's why we're probably going to wait until that merges and we can use that for mirroring debian/ubuntu
14:05:35 <frickler> yes, I'd second that
14:06:01 <SvenKieske> and which quay.io account are we going to use? I think there was a kolla account, right?
14:06:02 <mnasiadka> Ok then, any other CI failures not related to DockerHub?
14:06:15 <mnasiadka> SvenKieske: we have openstack.kolla org, I'd say we use that.
14:06:24 <SvenKieske> good :)
14:06:28 <frickler> mnasiadka: yes, jobs failing on raxflex
14:06:55 <mnasiadka> frickler: how are they failing?
14:07:00 <frickler> example at https://zuul.opendev.org/t/openstack/build/d7a108adc1c8491a92362bcb40b37670 I hope i can look into this later this week
14:07:13 <frickler> essentially: Error mounting /var/lib/docker: mount: /var/lib/docker: can't find LABEL=kolla.
14:07:32 <mnasiadka> Ah, I've seen that
14:07:37 <mnasiadka> thought it's some kernel bug
14:07:38 <SvenKieske> yep, I had roughly 5-10 of those the last days
14:07:39 <frickler> so something being different about the ephemeral volume. maybe there isn't any in that cloud
14:08:00 <frickler> usually these are retried because the failure happens in pre
14:08:12 <mnasiadka> well, we check if there is one - https://github.com/openstack/kolla/blob/master/roles/configure-ephemeral/tasks/main.yml
14:08:43 <frickler> if someone finds the cause for this, I won't mind, too ;)
14:08:44 <mnasiadka> either /dev/xvde or a disk with "ephemeral" label
14:09:06 <frickler> ah, it might be that /dev/xvde is swap instead
14:09:22 <mnasiadka> that makes things complicated
14:09:36 <mnasiadka> but we can always check for raxflex ;-)
14:09:49 <frickler> anyway, the data to debug should be in the logs
14:09:56 <SvenKieske> we really should not rely on special device names, labels are more stable
14:10:07 <mnasiadka> darmach68: want to have a look?
14:10:07 <frickler> if not, I can also hold a node if needed
14:10:12 <mnasiadka> SvenKieske: some cloud does /dev/xvde but no labels :)
14:10:18 <SvenKieske> but I never looked at that file
14:10:19 <darmach688> mnasiadka sure!
14:10:29 <mnasiadka> frickler: we have a volunteer
14:10:39 <frickler> cool
14:11:15 <SvenKieske> nice
14:12:09 <mnasiadka> ok then, let's move on
14:12:19 <mnasiadka> #topic Release tasks
14:12:41 <mnasiadka> bbezak: I think we have around two weeks to switch to Epoxy sources?
14:12:59 <bbezak> let me see
14:13:06 <mnasiadka> yup, R-17
14:13:10 <bbezak> R-17: Switch source images to current release¶
14:13:11 <bbezak> yeap
14:13:19 <opendevreview> Dawud proposed openstack/kolla-ansible master: Add size limits to Fluentd buffers  https://review.opendev.org/c/openstack/kolla-ansible/+/924359
14:13:20 <bbezak> Dec 02 - Dec 06
14:14:23 <mnasiadka> frickler: /dev/vdb has ephemeral0 label on raxflex
14:15:37 <mnasiadka> but for some reason when we create a filesystem with kolla label - it's not there for mounting
14:16:16 <mnasiadka> bbezak: willing to raise those next week? or do we need another volunteer?
14:18:27 <bbezak> source image changes mnasiadka ?
14:18:31 <mnasiadka> yup
14:18:34 <bbezak> willdo
14:18:37 <mnasiadka> there are three changes listed :)
14:18:42 <mnasiadka> goodie
14:18:54 <mnasiadka> #topic Current cycle planning
14:19:59 <mnasiadka> So, I started our WSGI mission with Gunicorn, but that's hard to implement because it requires changes in at least Nova - so probably not a good idea
14:20:12 <mnasiadka> so I went with uWSGI, although it's in maintenance mode - but Devstack and others are using it
14:20:20 <mnasiadka> And it seems to work now even with backend TLS
14:20:24 <mnasiadka> So the question is
14:20:40 <mnasiadka> do we want to support both apache/mod_wsgi and uwsgi for a cycle?
14:20:54 <SvenKieske> is it worth it to publish your findings wrt to WSGI to the ML as well? to get broader feedback, e.g. from nova folks?
14:20:58 <bbezak> I think we should? at least for keystone/horizon
14:21:05 <mnasiadka> SvenKieske: that's my plan
14:21:16 <mnasiadka> well, apache (or nginx) and uwsgi - yes
14:21:31 <mnasiadka> I'm asking if we should replace apache/mod_wsgi with uwsgi in Epoxy
14:21:37 <mnasiadka> or support both for one cycle
14:21:44 <frickler> there is some discussion to happen in the scope of the eventlet-rm team about which wsgi/asgi solution to use in the future
14:21:55 <frickler> so we might need to change again later
14:22:08 <frickler> support both for transition seems better to me
14:22:14 <parasitid> hi again, one simple question: is there a ref doc where i can find what are the conf options i can setup in a kolla-build.conf ? i didnt manage to find any on the docs.openstack.org site. thanks a lot
14:22:21 <SvenKieske> https://github.com/pallets/werkzeug is afaik a still maintained wsgi server, not sure if it's "better" though.
14:22:25 <mnasiadka> parasitid: we have a meeting now, please ask later
14:22:52 <mnasiadka> SvenKieske: I think I prefer to go with what others are doing, at least we see the same problems
14:23:12 <mnasiadka> At least for now :)
14:23:30 <mnasiadka> frickler: yeah, seen that, but it doesn't seem like it will be changed in Epoxy
14:23:40 <SvenKieske> sure, absolutely, would be nice if openinfra could converge on the same solution! but I would like something alive upstream :)
14:24:06 <SvenKieske> that said I don't know if uwsgi is just simply "finished", maybe? :)
14:24:30 <mnasiadka> anyway, so we go with both apache/mod_wsgi and uwsgi in Epoxy and deprecate apache/mod_wsgi approach so it's removed in F (unless we see some issues)
14:24:49 <mnasiadka> SvenKieske: You're trying to use the funny term "feature complete"? :)
14:25:20 <SvenKieske> yeah :) but looking at the c code of uwsgi I'm not so certain I would bet my life on it.
14:26:06 <mnasiadka> yes, I went with gunicorn since Skyline uses it, but then it has problems passing CLI args - I'll write about my adventure on ML and we can take it from there
14:26:43 <SvenKieske> nice
14:26:48 <mnasiadka> Any other new features/changes we need to discuss?
14:27:10 <SvenKieske> frickler: is there a link to the eventlet-rm team discussion about wsgi/asgi solution? was that only on the ML?
14:27:30 <mnasiadka> SvenKieske: there's a channel - #openstack-eventlet-removal
14:27:38 <frickler> SvenKieske: no, there is a (biweekly?) meeting
14:27:47 <mnasiadka> and meeting logs should be in the usual place
14:28:01 <SvenKieske> ah ty
14:28:26 <frickler> also some notes in https://etherpad.opendev.org/p/epoxy-eventlet-tracking
14:28:52 <mnasiadka> Ok, from another front - I think it would be good if somebody would focus on getting RMQ QueueManager stuff tested and validated
14:28:57 <mnasiadka> Matt Crees: alive?
14:29:57 <mnasiadka> Ok, I'll chase him internally - maybe he can do some work in that area
14:30:16 <frickler> that new ovs exporter project also looked interesting
14:30:17 <kevko> mnasiadka: there is need to share /dev/shm
14:30:20 <SvenKieske> yes, that would be nice, it should work now properly in containers
14:30:25 <kevko> Quemanager
14:30:29 <SvenKieske> if we have the above device^^
14:30:30 <kevko> for oslo services
14:30:48 <kevko> then ..it can work nice ..
14:30:49 <mnasiadka> kevko: I know, I'm just trying to find somebody else than you to test it out thoroughly :)
14:31:08 <mnasiadka> And I assume that's needed for the fanout queues in new RMQ world?
14:31:21 <kevko> mnasiadka: like i am not testing well ? :D
14:31:33 <kevko> mnasiadka: yep, correct ... fanouts ..
14:31:50 <mnasiadka> kevko: I'd prefer that we don't blame everything on you :)
14:32:11 <SvenKieske> yes, pretty certain we need the queue manager to get this correctly working
14:32:38 <SvenKieske> there was a ML report having issues with quorum queues, but there was not much information what the actual issue was - could've been just misconfiguration
14:33:01 <mnasiadka> Well, I've seen issues with quorum queues and ironic-neutron-agent
14:33:30 <mnasiadka> I don't know if that's rather a fanout queue, or not - in classical world it had autodelete on
14:33:42 <kevko> Haha, I don't recall you ever having to blame me for anything :D, as far as I know, everything works :)
14:33:58 <mnasiadka> and now it doesn't, so it's piling up queues and messages unless you delete those old queues without consumers
14:34:10 <mnasiadka> kevko: nobody's perfect :)
14:35:25 <mnasiadka> Ok then, let's move on
14:35:36 <mnasiadka> #topic Additional agenda (from whiteboard)
14:36:15 <mnasiadka> (SvenKieske 2024-11-13) reviews would be nice:
14:36:22 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/928949
14:36:47 <mnasiadka> https://review.opendev.org/q/+status:open+-is:WIP+uploader:kieske@osism.tech+label:Verified%252B1+-label:Workflow-1+is:mergeable
14:37:02 <mnasiadka> If anybody got some time, especially our quickstartguide needs to be changed since our arguments are reversed now (good first issue, if you are new to kolla): https://bugs.launchpad.net/kolla-ansible/+bug/2087920
14:37:16 <mnasiadka> darmach68: I know you complained over the quickstart guide recently, right?
14:37:20 <SvenKieske> ah I probably get around to fix the quick start guide today
14:37:35 <SvenKieske> but I won't complain if someone is faster :D
14:37:54 <SvenKieske> that list above is mostly backports I think
14:38:19 <mnasiadka> ok then, I'll just leave it there - I reapplied my +2 on the first patch from the list
14:38:28 <frickler> the issue darmach found was with the deploy guide? different bug
14:38:36 <mnasiadka> Ah, that's another thing
14:38:58 <mnasiadka> We were discussing that today, that probably those variables that we use in docs conf.py should land in deploy-guide conf.py as well
14:39:08 <mnasiadka> so maybe we need to move that to some common place and source it?
14:39:15 <frickler> +1
14:39:30 <mnasiadka> darmach688: you've got your answer
14:39:44 <mnasiadka> Next one from the whiteboard
14:39:45 <mnasiadka> (frickler 2024-11-27) transition stable/2023.1 to unmaintained?
14:39:48 <mnasiadka> https://review.opendev.org/c/openstack/releases/+/934490
14:40:00 <mnasiadka> I think all non-deployment projects already moved to unmaintained?
14:40:18 <frickler> well almost all, some still lagging
14:40:43 <mnasiadka> Do we need to do anything in Kolla like switch to use unmaintained sources?
14:40:53 <frickler> but a lot of stable/2023.1 branches are already gone
14:41:12 <frickler> so yes, those references would need to get switched likely like was done for zed?
14:41:25 <SvenKieske> yes, I think so, also in the docs, no?
14:41:29 <mnasiadka> Yes, so any volunteer to do the switch?
14:42:05 <frickler> a related question would be whether some of the older branches can be eoled? or who is volunteering to take care of them?
14:42:17 <mnasiadka> I'll do that again... darmach68 ?
14:42:25 <SvenKieske> #link https://review.opendev.org/c/openstack/kolla-ansible/+/927022
14:42:28 <mnasiadka> frickler: from my perspective yoga and zed can be EOLed
14:42:43 <mnasiadka> if anybody is there - it's his own fault ;-)
14:42:51 <SvenKieske> that's an example what needs to be fixed in the docs part at least
14:43:24 <SvenKieske> that as well: https://review.opendev.org/c/openstack/kolla-ansible/+/917421 (update .gitreview)
14:43:57 <SvenKieske> if you search for unmaintained you find quite some stuff :)
14:44:03 <mnasiadka> yup
14:44:15 <darmach> Yep, let it be me :D
14:44:21 <mnasiadka> ok then
14:44:23 <frickler> the gitreview update patch will be auto-created
14:44:29 <SvenKieske> drop upgrade testing as well
14:44:43 <SvenKieske> ah right
14:44:54 <frickler> so if there are no complaints, we could just go ahead with the release patch first
14:45:03 <mnasiadka> ok then, once we fix antelope and transition it to unmaintained - we can EOL yoga and zed
14:45:09 <frickler> and amend the unmaintained branches as needed afterwards?
14:45:13 <mnasiadka> frickler: fine by me
14:45:54 <mnasiadka> commented on the patch
14:46:03 <mnasiadka> ok then, let's go to open discussion
14:46:06 <mnasiadka> #topic Open discussion
14:46:12 <mnasiadka> Anybody anything?
14:46:15 <frickler> ah, you were faster than me ;)
14:46:23 <frickler> yes, one more thing
14:46:25 <mmalchuk> Certificates defaults
14:46:26 <mmalchuk> https://review.opendev.org/c/openstack/kolla-ansible/+/934514
14:46:30 <mmalchuk> please review
14:47:11 <frickler> cberendt (fellow kolla core) is running for foundation board, maybe someone wants to support his nomination https://openinfra.dev/a/community/members/7040
14:47:41 <mnasiadka> mmalchuk: I'm rather thinking about -1 than merging - I don't think people should be using that role for anything except development envs and testing
14:48:07 <mmalchuk> mnasiadka nice) why?
14:48:11 <mnasiadka> frickler, SvenKieske, bbezak, kevko what do you think?
14:48:25 <mnasiadka> because Kolla-Ansible should not be in the certificate management business :)
14:48:28 <SvenKieske> it _might_ be useful if you need to test some specific cert/CA settings
14:48:32 <mmalchuk> peoples already use this for production for a years)
14:48:40 <SvenKieske> but yeah, maybe there are better tools for that
14:48:54 <SvenKieske> you really shouldn't use this for production, honestly
14:49:21 <SvenKieske> use a proper CA at least, there are some projects on github etc which do this
14:49:40 <mmalchuk> SvenKieske but there is no other tools in Kolla for production)
14:50:25 <SvenKieske> you can always push your CA cert with external cert role, no?
14:50:39 <SvenKieske> e.g. something like this (no endorsement, there are others as well): https://github.com/smallstep/certificates
14:50:49 <kevko> mnasiadka: kolla-ansible should be cert manager ...but still ...everyone is using it :D :D :D
14:50:57 <kevko> *shouldn't be
14:51:05 <mmalchuk> I do that, only CA and use role to generate endlevel certs
14:51:41 <opendevreview> Michal Nasiadka proposed openstack/kolla-ansible master: nova: Add support for using uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/935975
14:51:45 <kevko> I am ok with it ...it's just movement values to variables ..that's all
14:52:04 <mnasiadka> I think we discussed some PTGs ago to support things like smallstep
14:52:06 <mmalchuk> yep
14:52:36 <SvenKieske> yes, I mean of course people will be using test CAs in prod, it doesn't matter how many warnings you put into docs :D
14:52:52 <mnasiadka> Yes, I don't know if we should make it easier though :)
14:53:05 <mnasiadka> And I don't know if I like the command: style of that role ;-)
14:53:07 <mmalchuk> the bugreport is open, this change only PartialBug so lets merg it and add smalstep or whatever and close the bug
14:53:47 <mnasiadka> It's not a bug
14:53:55 <mnasiadka> It's by design
14:54:06 <mmalchuk> design with bug)))
14:54:10 <kevko> Well, if it will be under ansible/roles ...from my perspective it can be used by user ...we also providing kolla-ansible certificates command ...so if we don't want from users to use it ...remove it and place into kolla collections or somewhere ...or something similar as we did for ceph
14:55:03 <mmalchuk> yes, it in the K-A but can't be used and configured
14:55:15 <mnasiadka> That was my comment long time ago when we added that certificates role - but obviously it makes life easier when doing a development/test deployment with TLS
14:55:15 <mmalchuk> so this is an issue
14:55:44 <mnasiadka> Issue why?
14:56:11 <mnasiadka> Tell us why do you need to set those variables to do a K-A deployment for testing
14:56:22 <mmalchuk> certs generated with wrong defaults
14:56:28 <mmalchuk> at least days
14:56:35 <SvenKieske> from a users perspective I can see why you would want it. from a dev perspective I would argue to make it harder to use and replace by proper CA integration. you can use your CA api for configuring all this stuff, it's outside of kollas domain imho.
14:56:36 <mmalchuk> at least days of validity
14:57:04 <SvenKieske> isn't the current validity a year or something?
14:57:46 <SvenKieske> maybe I don't understand the validity issue.
14:58:00 <mnasiadka> Me neither
14:58:06 <mmalchuk> after an year peoples forgot about self-signed cert and get an issue in api/browser again
14:58:32 <frickler> after a year a test environment is broken anyway
14:58:37 <mmalchuk> some of them want certs in dev for 10 years for example
14:58:37 <mnasiadka> you've got year long development environments?
14:58:56 <mmalchuk> yep
14:59:06 <mmalchuk> we still have ussuri )))
14:59:06 <SvenKieske> so either you automate your CA, (you need to control client CA deployment anyway I guess)
14:59:26 <mmalchuk> on Centos7 )))
14:59:27 <SvenKieske> I mean changing validity is orthogonal to make it configurable
14:59:31 <darmach> I'd assume that long existing dev/test env should already be rebuilt ;)
14:59:51 <SvenKieske> we could propose to set validity to seven days only, and hardcode that, to make clear this is only for dev mode? ;)
15:00:07 <frickler> SvenKieske: +1
15:00:18 <mmalchuk> lol
15:00:24 <darmach> SvenKieske +1 for harcore solutions!
15:00:31 <mnasiadka> Yeah, probably that's what we should rather do
15:00:49 <SvenKieske> poor users using this in prod xD
15:01:04 <mnasiadka> maybe 14 or 30 so we're not that hard on people - but that shouldn't be a year
15:01:15 <mmalchuk> lets drop the certificates role at all! and ask users to use their own CA
15:01:24 <mmalchuk> this would be hardcore)
15:01:29 <mnasiadka> We do that in the docs, don't we?
15:01:41 <SvenKieske> I think it's really good to have a cert stub like this for easy testing
15:01:44 <mmalchuk> no one reads the docs)
15:02:10 <mnasiadka> I'm happy to move that role outside of kolla-ansible delivered roles and just give a bash script to create certs for dev env with TLS
15:02:12 <SvenKieske> or we go full in and implement a proper CA even for testing, that would be better, but nobody got time for that I guess?
15:02:41 <SvenKieske> we at least need something to test TLS in CI
15:02:46 <mmalchuk> oh cool. lets rewrite ansible to bash)
15:03:04 <kevko> haha ..let's do that :D 7 days is fine :D
15:03:30 <darmach> mmalchuk I started recently, and found out that half is outdated, the other half doesn't template/render properly ;)
15:03:31 <kevko> we will see how many bugreports we will receive :D
15:03:38 <SvenKieske> anyway we are over time for the meeting
15:03:47 <mnasiadka> yup
15:03:56 <kevko> Okay, can anybody advise me I've asked before meeting ?
15:04:07 <kevko> do anybody know if i can  1. have external public network with dns_domain set - shared, 2. zone for that network => Can another tenant create VM on this network and it will be automatically DNS created ?
15:04:14 <mnasiadka> so the consensus is let's leave it as it is or make it harder for users to use ;-)
15:04:56 <frickler> kevko: no, external networks are intentionally excluded
15:05:08 <mmalchuk> be prepare for bugreports)
15:05:44 <kevko> frickler: Hmm, I had that feeling ...
15:05:55 <kevko> frickler: what about zone transfer request ?
15:06:29 <mnasiadka> ok then, let's finish the meeting
15:06:30 <mnasiadka> #endmeeting