| opendevreview | OpenStack Proposal Bot proposed openstack/magnum master: Imported Translations from Zanata https://review.opendev.org/c/openstack/magnum/+/871939 | 04:25 |
|---|---|---|
| jakeyip | I've never tried that TBH | 08:53 |
| jakeyip | hi all | 08:53 |
| jakeyip | please add to agenda | 08:54 |
| jakeyip | please add to agenda https://etherpad.opendev.org/p/magnum-weekly-meeting | 08:54 |
| dalees | hi jakeyip, all | 08:59 |
| jakeyip | hi dalees :) | 08:59 |
| jakeyip | #startmeeting magnum | 09:00 |
| opendevmeet | Meeting started Wed Feb 15 09:00:34 2023 UTC and is due to finish in 60 minutes. The chair is jakeyip. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:00 |
| opendevmeet | The meeting name has been set to 'magnum' | 09:00 |
| jakeyip | #topic Roll Call | 09:00 |
| jakeyip | o/ | 09:00 |
| dalees | o/ | 09:01 |
| travissoto | o/ | 09:01 |
| jakeyip | #link https://etherpad.opendev.org/p/magnum-weekly-meeting | 09:02 |
| jakeyip | Please feel free to populate the agenda | 09:02 |
| jakeyip | hi travissoto | 09:03 |
| jakeyip | thanks everyone for coming to the meeting. feel free to join in at anytime. | 09:03 |
| jakeyip | #topic PTG | 09:03 |
| travissoto | hi all | 09:04 |
| jakeyip | There are two PTGs this time, (1) Vitual PTG (March 27-31) (2) OpenInfra Summit + PTG (June 13-15) | 09:04 |
| jakeyip | does everyone have a preference for PTG? | 09:05 |
| jakeyip | unfortunately I probably will not be able to make it to OpenInfra | 09:05 |
| jakeyip | if there is no hard preference I may book something for March and see if there will be interest closer to the date | 09:06 |
| dalees | likewise, virtual is preferred this time for me. | 09:06 |
| travissoto | +1 | 09:06 |
| jakeyip | ok | 09:07 |
| jakeyip | #action jakeyip to book Virtual PTG | 09:07 |
| jakeyip | #topic Antelope supported versions | 09:07 |
| jakeyip | Thanks everyone for the patches to make FCOS 36-37 work | 09:08 |
| jakeyip | and also k8s v1.24 | 09:08 |
| jakeyip | A common issue for user is that they are unsure which versions of FCOS / K8S is supported. For that I have recently fixed up the docs to reflect that | 09:09 |
| jakeyip | #link https://docs.openstack.org/magnum/latest/user/#supported-versions | 09:09 |
| jakeyip | I would like to propose that we say FCOS 36/37 + k8s v1.24 is supported for this cycle. | 09:10 |
| jakeyip | what does everyone think? | 09:10 |
| dalees | I've also passed conformance for 1.25, and had 1.26 mostly running (but I have not reviewed kube-system pod versions yet). | 09:10 |
| dalees | sounds good, 1.24 is still supported k8s version. | 09:10 |
| jakeyip | dalees: does default labels work ? | 09:11 |
| dalees | jakeyip: unlikely, that was the other topic I'd like to discuss. We bump a large number of things and the defaults are way out of date now (calico etc) | 09:11 |
| jakeyip | dalees: yeah that is a problem. lots of discussion needed there :) | 09:12 |
| jakeyip | ok if we are in agreement let's just target 1.24 and 1.25 as stretch goal ;) | 09:12 |
| jakeyip | #info Antelope supported version FCOS 36/37 and Kubernetes v1.24 | 09:13 |
| dalees | maybe we can share (or update) working template labels if we have the 1.24 locked in. it's hard to know to update the default without the version (1.24) set in place. | 09:13 |
| jakeyip | we can target tests for these versions, and we can have labels that work possibly out of the box (later discussion) | 09:13 |
| jakeyip | great :) | 09:13 |
| jakeyip | #topic Deprecation | 09:14 |
| jakeyip | As Antelope is about to come to an end, I am in a bit of a hurry to mark things as deprecated this cycle, so to allow them to be removed in 2 cycles' time | 09:15 |
| jakeyip | #topic Deprecate Fedora Atomic for Kubernetes | 09:16 |
| jakeyip | #link https://review.opendev.org/c/openstack/magnum/+/833949 I see a few +1, I think we can get this in this cycle. Thanks dalees :) | 09:16 |
| jakeyip | #topic Deprecate Swarm | 09:16 |
| jakeyip | is anybody still using swarm? or not? | 09:16 |
| dalees | not us. | 09:17 |
| jakeyip | we are not using Swarm at all, and I'm not even sure it works. | 09:17 |
| jakeyip | OK if there is someone using Swarm please feel free to email me. If not I will drop a mail on the ML to see who is using and wants to take up maintenance | 09:18 |
| jakeyip | #action jakeyip Propose deprecation of Swarm to ML | 09:18 |
| dalees | +1 needs a mailing list post, and then propose removal if it's not relevant anymore. | 09:19 |
| jakeyip | :) | 09:19 |
| jakeyip | if we can get Fedora Atomic out there is lots of code we can remove | 09:19 |
| jakeyip | #topic python-magnumclient intermittent failures after tox4 | 09:20 |
| jakeyip | So I tried updating to tox4 format, but weirdly I am getting intermittent failures running `tox -e py38` with those changes. | 09:21 |
| jakeyip | it fails in check too | 09:21 |
| jakeyip | we have patches stuck because of this. if someone can help it'll be great | 09:22 |
| jakeyip | alright we went through the previous items pretty quickly, are there questions for those items, dalees / travissoto ? | 09:23 |
| dalees | i'll see if i can look into those failures with magnumclient, the "intermittent" part is concerning. | 09:24 |
| travissoto | no not from me at this stage :) | 09:24 |
| jakeyip | thanks dalees | 09:24 |
| jakeyip | dalees: do you want to discuss prometheus helm chart now? | 09:25 |
| dalees | yeah, sure. | 09:26 |
| jakeyip | #topic Prometheus helm charts | 09:26 |
| dalees | so the prometheus/grafana stack is installed into kube-system namespace with helm if monitoring_enabled is set. | 09:26 |
| dalees | and this breaks in 1.22 | 09:27 |
| dalees | we(actually, travissoto ) replaced the helm charts with the newer, completely different ones from kube-prometheus-stack. | 09:27 |
| dalees | do others want or use these, or should we keep this patch local and remove the complexity from Magnum? | 09:28 |
| jakeyip | I think I tried the default one and gave up :) | 09:29 |
| jakeyip | dalees: do you install it for all your users? | 09:30 |
| dalees | our templates enable it by default yes, but many turn it off and install their own monitoring stacks. | 09:30 |
| jakeyip | yeah ok | 09:31 |
| jakeyip | we don't install it by default IIRC and I suspect users might prefer their own | 09:32 |
| dalees | as we refactor to CAPI, we're going to consider if and how we keep it. It's a big job to keep it maintained in Magnum codebase. | 09:32 |
| jakeyip | of cos having more things work out of the box is great for users, but maintaining them up to date is an issue | 09:33 |
| jakeyip | and the more is in the codebase, the more we are responsible for testing | 09:33 |
| jakeyip | we don't have good test for that now (?) so merging changes will be difficult without tests | 09:34 |
| dalees | yeah, with k8s 1.24 as supported i suspect that won't work out of the box. | 09:35 |
| jakeyip | I propose that we should remove it if it is broken. What does everyone think? | 09:35 |
| travissoto | agree better to remove it | 09:35 |
| dalees | ok by me | 09:36 |
| dalees | lets confirm it's broken first. | 09:36 |
| dalees | (it is for us, but i want to be sure it's not our local patches) | 09:37 |
| jakeyip | OK. can you help to confirm in devstack and send up a change to remove it if it is? | 09:37 |
| jakeyip | I may take a look later too | 09:37 |
| dalees | ok | 09:38 |
| jakeyip | #action dalees / travissoto to confirm prometheus helm chart is broken, and propose patch to remove if it is | 09:39 |
| jakeyip | #topic Container Labels | 09:39 |
| jakeyip | big topic ;) | 09:39 |
| jakeyip | I guess executive summary is: default labels may be broken, but updating them may break existing cluster templates that do not set them | 09:40 |
| jakeyip | what are our options? | 09:40 |
| jakeyip | oh oh | 09:42 |
| dalees | yep, that's pretty difficult. making a template that relies on defaults that may change isn't a great experience. | 09:43 |
| dalees | we've a similar issue with manifests, if someone updates the calico manifests for version v1.23 and we're still allowing users to create calico v1.13 their new clusters all break. | 09:44 |
| dalees | I've resolved this by copying all templates that change and picking them up with version matches (as seen in https://github.com/openstack/magnum/blob/master/magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml#L1424 ) | 09:46 |
| jakeyip | look on the bright side, new cluster breaking is better than current cluster breaking | 09:47 |
| dalees | back to labels topic though - i think defining as many labels as possible in a template helps, which is what we end up doing. it means the defaults don't apply. | 09:47 |
| jakeyip | that's what we do too because the defaults are just too old | 09:48 |
| dalees | so who has the problem of user templates breaking? do we just run with it and update them to match the current k8s (1.24 right now)? and produce example templates that can be published with little chance of breaking? | 09:49 |
| jakeyip | I did have our organisation templates broken before, that's why I learnt to pin as many labels as possible | 09:50 |
| dalees | you mentioned breaking existing clusters - how are these labels ever re-applied to a running cluster? the upgrade or scaling process doesn't do it (if we're talking kube-system container images). Existing Heat stacks stay the same. | 09:51 |
| jakeyip | yeah sorry I mean existing cluster _templates_, I might have typo | 09:51 |
| dalees | ah ok; just checking I understood properly | 09:52 |
| dalees | this type of problem may not go away with CAPI. we still need some concept of cluster templates. | 09:53 |
| jakeyip | yeah, I feel the least disruptive is to leave the defaults alone and document what works for the current versions | 09:55 |
| dalees | but then you sacrifice the "works out of the box" experience, if that is the goal. | 09:55 |
| jakeyip | updating the labels in code is an impossible task. we can push it to latest in Antelope, but by the time an organisation installs / upgrade to Antelope they will be out of date already | 09:56 |
| dalees | you could remove all defaults and force them to be specified in template labels :) | 09:56 |
| jakeyip | :) | 09:56 |
| jakeyip | for CAPI? :) | 09:56 |
| dalees | it's worth considering yeah. | 09:57 |
| dalees | then you only maintain versions in once place, and they match the k8s version | 09:57 |
| jakeyip | yeah agree much nicer | 09:58 |
| jakeyip | I guess what we can do better is document it. I've heard complains :) | 09:58 |
| jakeyip | we are almost out of time. any other topic? | 09:58 |
| dalees | keen to hear others' ideas on the topic, who aren't in the meeting but involved in Magnum. | 09:59 |
| dalees | I've got some for another week, but they can wait. thanks for the discussion | 09:59 |
| jakeyip | me too. let's hold this regularly and more may join | 10:00 |
| jakeyip | Thanks dalees and travissoto for coming | 10:00 |
| jakeyip | #endmeeting | 10:01 |
| opendevmeet | Meeting ended Wed Feb 15 10:01:03 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 10:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/magnum/2023/magnum.2023-02-15-09.00.html | 10:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/magnum/2023/magnum.2023-02-15-09.00.txt | 10:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/magnum/2023/magnum.2023-02-15-09.00.log.html | 10:01 |
| jakeyip | dalees / travissoto : how many people from catalyst are working on Magnum? | 10:07 |
| dalees | jakeyip: the two of us currently, with another learning the internals and a couple of others in the immediate team. We cover other services also, so not just Magnum. | 10:11 |
| dalees | jakeyip: how about your org? | 10:12 |
| jakeyip | dalees: generally only me. yes similarly I also help in other services. | 10:15 |
| opendevreview | Matthew Heler proposed openstack/magnum master: Support multi AZ for k8s multi masters https://review.opendev.org/c/openstack/magnum/+/714347 | 10:51 |
| supamatt | dalees: do you a wip patch of those prometheus changes available somewhere? | 11:21 |
| guilhermesp_____ | hey jakeyip just saw your reply on the conformance email. Yeah i think that could be PSP in fact. kube-apiserver fails to start with a rancher-1.25.* image and magnum master | 12:35 |
| guilhermesp_____ | Feb 14 20:24:06 k8s-cluster-dgpwfkugdna5-master-0 conmon[119164]: E0214 20:24:06.615919 1 run.go:74] "command failed" err="admission-control plugin \"PodSecurityPolicy\" is unknown" | 12:35 |
| guilhermesp_____ | if you want a full trace of the logs, dont hesitate i can share them all :) | 12:37 |
| mnasiadka | oops, forgot about the meeting | 13:39 |
| opendevreview | Tyler proposed openstack/magnum master: Update devstack plugin with capi management https://review.opendev.org/c/openstack/magnum/+/872755 | 13:47 |
| opendevreview | Tyler proposed openstack/magnum master: Update devstack plugin with capi management https://review.opendev.org/c/openstack/magnum/+/872755 | 16:58 |
| supamatt | guilhermesp_____: I have k8s 1.26.1 working, you need to remove PodSecurity from the admissision list. This can be done with a label. | 17:23 |
| opendevreview | Tyler proposed openstack/magnum-tempest-plugin master: DNM: WIP Get tests passing on cluster-api https://review.opendev.org/c/openstack/magnum-tempest-plugin/+/872759 | 17:38 |
| opendevreview | Tyler proposed openstack/magnum master: Update devstack plugin with capi management https://review.opendev.org/c/openstack/magnum/+/872755 | 17:43 |
| opendevreview | Elod Illes proposed openstack/python-magnumclient master: DNM: dummy change to test gate health https://review.opendev.org/c/openstack/python-magnumclient/+/874014 | 19:37 |
| dalees | guilhermesp_____: supamatt: Ah! I hadn't realized we'd defaulted our admission controller to remove PodSecurityPolicy which is allowing 1.25 to function. Goes back to the labels discussion earlier in meeting - I'll propose a changeset to Magnum to remove it, but it may be something that needs to be specified per k8s version (1.20 clusters might | 20:36 |
| dalees | like it, and 1.25 cannot have it). | 20:36 |
| dalees | supamatt: I'll see what we can do to share the prometheus changes, happy to. | 20:37 |
| opendevreview | Dale Smith proposed openstack/magnum master: Remove PodSecurityPolicy from default admission controller list https://review.opendev.org/c/openstack/magnum/+/874031 | 20:58 |
| dalees | ^ created this for discussion. It's directly related to the meeting discussion on default labels - perhaps publishing example magnum templates for each k8s version is the way to go, instead. | 20:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!