| *** ttsiouts has joined #openstack-containers | 00:38 | |
| *** ttsiouts has quit IRC | 00:47 | |
| *** ttsiouts has joined #openstack-containers | 00:48 | |
| *** ttsiouts has quit IRC | 00:52 | |
| *** ricolin has joined #openstack-containers | 01:02 | |
| *** hongbin has joined #openstack-containers | 01:33 | |
| *** ricolin_ has joined #openstack-containers | 01:50 | |
| *** ricolin has quit IRC | 01:50 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: [fedora_atomic] Support auto healing for k8s https://review.openstack.org/631378 | 02:18 |
|---|---|---|
| *** hongbin has quit IRC | 02:35 | |
| *** hongbin has joined #openstack-containers | 02:38 | |
| *** hongbin has quit IRC | 02:49 | |
| *** hongbin has joined #openstack-containers | 02:51 | |
| *** ykarel|away has joined #openstack-containers | 02:51 | |
| flwang | jakeyip: ping | 03:01 |
| flwang | jakeyip: could you please help review https://review.openstack.org/#/c/651027/ ? thanks | 03:01 |
| *** ramishra has joined #openstack-containers | 03:59 | |
| *** udesale has joined #openstack-containers | 04:04 | |
| *** hongbin has quit IRC | 04:05 | |
| *** ykarel|away is now known as ykarel | 04:07 | |
| *** ykarel has quit IRC | 04:42 | |
| *** ykarel has joined #openstack-containers | 04:56 | |
| *** sidx64 has joined #openstack-containers | 05:08 | |
| *** udesale has quit IRC | 05:52 | |
| *** ykarel is now known as ykarel|afk | 05:58 | |
| *** ricolin_ has quit IRC | 05:58 | |
| *** ykarel|afk has quit IRC | 06:02 | |
| *** ykarel|afk has joined #openstack-containers | 06:13 | |
| *** ricolin has joined #openstack-containers | 06:22 | |
| *** anyrude10 has joined #openstack-containers | 06:25 | |
| anyrude10 | Hi Team, Is there amy workaround for the bug 1809254, https://bugs.launchpad.net/ubuntu/+source/magnum/+bug/1809254 | 06:26 |
| openstack | Launchpad bug 1809254 in magnum (Ubuntu) "Cannot create kubernetes cluster with tls_disabled" [Undecided,Confirmed] | 06:26 |
| anyrude10 | Thanks | 06:29 |
| *** pcaruana has joined #openstack-containers | 06:30 | |
| *** udesale has joined #openstack-containers | 06:33 | |
| *** ykarel|afk is now known as ykarel | 06:35 | |
| *** ivve has joined #openstack-containers | 07:06 | |
| *** mgoddard has joined #openstack-containers | 07:10 | |
| *** sidx64_ has joined #openstack-containers | 07:12 | |
| *** sidx64 has quit IRC | 07:13 | |
| *** yankcrime has quit IRC | 07:15 | |
| *** sidx64 has joined #openstack-containers | 07:16 | |
| *** sidx64_ has quit IRC | 07:17 | |
| *** alisanhaji has joined #openstack-containers | 07:25 | |
| *** gsimondon has joined #openstack-containers | 07:29 | |
| *** ykarel is now known as ykarel|lunch | 07:31 | |
| *** sidx64 has quit IRC | 07:43 | |
| *** rcernin has quit IRC | 08:01 | |
| *** ttsiouts has joined #openstack-containers | 08:05 | |
| *** jaewook_oh has joined #openstack-containers | 08:05 | |
| *** johanssone has quit IRC | 08:18 | |
| *** flwang1 has joined #openstack-containers | 08:19 | |
| *** johanssone has joined #openstack-containers | 08:24 | |
| *** sidx64 has joined #openstack-containers | 08:28 | |
| *** yankcrime has joined #openstack-containers | 08:33 | |
| *** ttsiouts has quit IRC | 08:37 | |
| *** ttsiouts has joined #openstack-containers | 08:37 | |
| *** ttsiouts has quit IRC | 08:40 | |
| *** ttsiouts has joined #openstack-containers | 08:40 | |
| *** ykarel|lunch is now known as ykarel | 08:48 | |
| *** jaewook_oh has quit IRC | 08:57 | |
| *** flwang1 has quit IRC | 09:33 | |
| *** livelace has joined #openstack-containers | 09:33 | |
| *** sidx64 has quit IRC | 09:34 | |
| *** flwang1 has joined #openstack-containers | 09:34 | |
| flwang1 | dioguerra: hello, is Thomas around? | 09:34 |
| *** sidx64 has joined #openstack-containers | 09:45 | |
| *** yolanda_ has quit IRC | 09:47 | |
| *** sidx64 has quit IRC | 09:51 | |
| flwang1 | strigazi: could you please ask THomas login this channel? | 09:53 |
| *** sidx64 has joined #openstack-containers | 09:55 | |
| dioguerra | flwangl: yes he is, we going for lunch now. Can we talk in 1h? | 09:57 |
| *** udesale has quit IRC | 09:57 | |
| *** udesale has joined #openstack-containers | 09:58 | |
| dioguerra | flwang: The problem with the master node is also happening to me. | 09:58 |
| flwang1 | dioguerra: sure | 10:01 |
| flwang1 | i will wait until you guys back | 10:02 |
| flwang1 | thanks | 10:02 |
| anyrude10 | Hi Team, I am facing etcd issue in swarm-docker. Can someone please help https://ask.openstack.org/en/question/120717/swarm-magnum/ | 10:15 |
| *** ttsiouts has quit IRC | 10:25 | |
| *** ttsiouts has joined #openstack-containers | 10:26 | |
| *** ttsiouts has quit IRC | 10:30 | |
| *** sidx64 has quit IRC | 10:41 | |
| *** ykarel is now known as ykarel|afk | 10:41 | |
| *** ttsiouts has joined #openstack-containers | 10:46 | |
| flwang1 | dioguerra: when you test the auto scaling/healing patch, did you based on the latest code with node group support? | 10:52 |
| brtknr | flwang1: have you tried another container runtime with magnum? | 10:53 |
| brtknr | e.g. cri-o | 10:53 |
| flwang1 | brtknr: no | 10:53 |
| flwang1 | my current focus is the auto scaling/healing feature and the rolling upgrade feature | 10:54 |
| flwang1 | after that, i will do a big refactor for magnum ui | 10:54 |
| flwang1 | and then probably Fedora CoreOS 30 and containerize K8S master nodes | 10:55 |
| flwang1 | basically the plan i mentioned in my PTL nomination letter | 10:55 |
| *** livelace has quit IRC | 10:56 | |
| *** udesale has quit IRC | 10:56 | |
| *** sidx64 has joined #openstack-containers | 11:05 | |
| *** ykarel|afk is now known as ykarel | 11:12 | |
| *** sidx64 has quit IRC | 11:15 | |
| dioguerra | flwang: thomas is joining soon | 11:16 |
| *** thartland has joined #openstack-containers | 11:17 | |
| dioguerra | I think the scaling up/down i tested it arround commit a6c8c399e926ea42d52fc5ebf7715b74999a404c | 11:17 |
| thartland | Hi | 11:17 |
| flwang1 | thartland: hi | 11:18 |
| flwang1 | i'm pretty sure it worked before | 11:18 |
| dioguerra | Yes it did | 11:18 |
| flwang1 | but now after rebased on the node group patches | 11:19 |
| flwang1 | it's getting weird which we need to fix | 11:19 |
| dioguerra | We think it's rebuilding the master node because of the Security rules | 11:19 |
| flwang1 | what's the security rules you're taking about? | 11:20 |
| flwang1 | adding security rule from master to minion? | 11:20 |
| *** sidx64 has joined #openstack-containers | 11:21 | |
| flwang1 | do you mean this one https://review.openstack.org/#/c/647942/ ? | 11:21 |
| dioguerra | I think it was this one? 1f5dc1aa91f145b8554deda3fed7265d33b3cb22 | 11:22 |
| dioguerra | Yes that one | 11:22 |
| flwang1 | thartland: btw, i also got error when doing scale down http://paste.openstack.org/show/749051/ | 11:22 |
| flwang1 | dioguerra: did you try to remove the change? | 11:23 |
| dioguerra | not yet | 11:23 |
| flwang1 | dioguerra: did you try scale down? | 11:24 |
| flwang1 | now i got http://paste.openstack.org/show/749051/ when doing scale down | 11:24 |
| dioguerra | No, it breaks on scale up and the cluster is lost | 11:25 |
| dioguerra | is your coreDNS up? | 11:26 |
| *** sidx64 has quit IRC | 11:26 | |
| dioguerra | or is that the minion name on openstack? | 11:26 |
| flwang1 | dioguerra: on my testing, the cluster will be back finally | 11:26 |
| thartland | flwang1: what are the values in the refs_map output_value of the minions stack? It should match one of the values printed in that error. openstack stack show cluster-kfr4yt3rm3mx-kube_minions-juk4gy42g2wo | 11:27 |
| flwang1 | because the master is being rebuilt so it takes some time | 11:27 |
| *** sidx64 has joined #openstack-containers | 11:27 | |
| flwang1 | thartland: http://paste.openstack.org/show/749052/ | 11:28 |
| thartland | flwang1: Ah, it's using the MachineID and the dashes are missing from the ID in the version kubernetes has | 11:29 |
| flwang1 | thartland: yes | 11:30 |
| flwang1 | thartland: MachineID:17578f36b3be4512926eced1a1c69eb3 ProviderID:openstack:///17578f36-b3be-4512-926e-ced1a1c69eb3 | 11:31 |
| flwang1 | it's very interesting | 11:31 |
| flwang1 | but the providerID is using the 'correct' format | 11:31 |
| flwang1 | how is the MachineID generated? | 11:32 |
| thartland | Not sure, there is also systemUUID which has the dashed but is capitalised e.g systemUUID: DE82824B-3E35-4214-970D-7C85B5F3C97A | 11:32 |
| flwang1 | thartland: ok, i think at least we need to use the format with dashes, because that's the format Nova is using | 11:35 |
| flwang1 | dioguerra: as for your testing, after the master rebuilt, did you see any other issue? | 11:36 |
| dioguerra | master rebuilds but i cannot contact the cluster anymore. | 11:38 |
| flwang1 | ok, that's not my case anyway | 11:38 |
| thartland | flwang1: the autoscaler already imports a uuid package so I can use that to parse the uuid and it return it in the correct format | 11:39 |
| flwang1 | dioguerra: could you pls test it again without the security group rules? | 11:40 |
| flwang1 | thartland: yep, please do | 11:40 |
| dioguerra | already on int | 11:42 |
| dioguerra | it | 11:42 |
| flwang1 | dioguerra: awesome, really appreciate it | 11:45 |
| flwang1 | thartland: should this function be revisited https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/magnum/magnum_manager_heat.go#L184 ? | 11:45 |
| flwang1 | the patch has been merged as you may know | 11:45 |
| thartland | flwang1: Possibly, I'm not sure if the autoscaler should get ahead of the release version of magnum but I can take a look at how it would eventually work | 11:49 |
| flwang1 | thartland: that would be great | 11:51 |
| flwang1 | thartland: i was told current patch will be in v1.14.1 | 11:51 |
| flwang1 | but now we're hosting the autoscaler image on openstackmagnum dockerhub repo | 11:51 |
| flwang1 | so it's not such rush | 11:51 |
| flwang1 | i mean we don't have to be in the v1.14.1 | 11:51 |
| thartland | flwang1: Once nodegroups and cluster resize are released I will start on a new magnum_manager interface to use those APIs and use all the new changes, and backport what makes sense to the old manager | 11:52 |
| flwang1 | we can release it on openstackmagnum repo firstly/temporarily | 11:52 |
| flwang1 | thartland: the cluster resize api has been released | 11:52 |
| flwang1 | and i have done all the work in gophercloud | 11:52 |
| flwang1 | you can start to integrate it now | 11:52 |
| flwang1 | and i even add the apiversions support for magnum in gophercloud, so that you can check the microversion to support both resize and the old way (magnum+heat) | 11:53 |
| flwang1 | thartland: ^ | 11:53 |
| thartland | flwang1: I saw the changes in gophercloud, thanks for those | 11:54 |
| flwang1 | thartland: thank you! | 11:55 |
| thartland | I have to leave for a few hours now, I'll be back later | 11:55 |
| flwang1 | thartland: ttyl | 11:55 |
| flwang1 | dioguerra: i have to offline, it's 11:56pm here | 11:56 |
| flwang1 | dioguerra: will you join today's weekly meeting? | 11:56 |
| *** sidx64 has quit IRC | 11:59 | |
| flwang1 | dioguerra: i have to go, please leave message based on your test result, thank you very much | 12:00 |
| *** ttsiouts has quit IRC | 12:00 | |
| *** jmlowe has quit IRC | 12:01 | |
| *** ttsiouts has joined #openstack-containers | 12:01 | |
| *** jmlowe has joined #openstack-containers | 12:02 | |
| dioguerra | flwangl: if i don't forget again | 12:12 |
| *** sidx64 has joined #openstack-containers | 12:17 | |
| *** livelace has joined #openstack-containers | 12:20 | |
| dioguerra | flwang: reverting 31c82625d6cae5b9cc8fae6f09c9107818dee9b7 does not work | 12:25 |
| *** livelace has quit IRC | 12:32 | |
| *** gsimondo1 has joined #openstack-containers | 12:42 | |
| *** openstackgerrit has quit IRC | 12:44 | |
| *** gsimondon has quit IRC | 12:44 | |
| *** ykarel is now known as ykarel|afk | 13:02 | |
| *** sidx64 has quit IRC | 13:05 | |
| *** ykarel|afk has quit IRC | 13:09 | |
| *** ykarel has joined #openstack-containers | 13:32 | |
| *** udesale has joined #openstack-containers | 13:36 | |
| *** lpetrut has joined #openstack-containers | 13:42 | |
| *** lpetrut has quit IRC | 14:04 | |
| *** ttsiouts has quit IRC | 14:06 | |
| *** ttsiouts has joined #openstack-containers | 14:07 | |
| *** ttsiouts has quit IRC | 14:09 | |
| *** ttsiouts has joined #openstack-containers | 14:09 | |
| brtknr | flwang1: What does auto-healing do which is distinct from auto-scaling? | 14:27 |
| *** livelace has joined #openstack-containers | 14:42 | |
| *** lpetrut has joined #openstack-containers | 14:48 | |
| *** ttsiouts has quit IRC | 15:01 | |
| *** ttsiouts has joined #openstack-containers | 15:02 | |
| *** ttsiouts has quit IRC | 15:05 | |
| *** ttsiouts has joined #openstack-containers | 15:06 | |
| *** sapd1_x has joined #openstack-containers | 15:22 | |
| *** sapd1_x has quit IRC | 15:27 | |
| *** lpetrut has quit IRC | 15:29 | |
| *** ivve has quit IRC | 15:32 | |
| *** udesale has quit IRC | 15:37 | |
| *** gsimondo1 has quit IRC | 15:42 | |
| *** ttsiouts has quit IRC | 15:56 | |
| *** ttsiouts has joined #openstack-containers | 15:57 | |
| dioguerra | brtknr: NPD checks logs for erros and tags nodes with a specific condition DRAINO: will consider de node not healthy if defined conditions are met and Drains and later evicts pods CA: creates new nodes if there are pods in pending state of creation | 16:00 |
| *** ttsiouts has quit IRC | 16:01 | |
| *** livelace has quit IRC | 16:06 | |
| *** ykarel is now known as ykarel|away | 17:12 | |
| *** ykarel|away has quit IRC | 17:17 | |
| *** ramishra has quit IRC | 17:19 | |
| *** livelace has joined #openstack-containers | 17:32 | |
| *** gmann is now known as gmann_afk | 17:40 | |
| *** sidx64 has joined #openstack-containers | 18:00 | |
| *** ricolin has quit IRC | 18:04 | |
| *** sidx64_ has joined #openstack-containers | 18:06 | |
| *** sidx64 has quit IRC | 18:06 | |
| *** alisanhaji has quit IRC | 18:16 | |
| *** gmann_afk is now known as gmann | 18:21 | |
| *** lpetrut has joined #openstack-containers | 18:25 | |
| flwang1 | dioguerra: thanks for the feedback | 18:26 |
| *** lpetrut has quit IRC | 18:29 | |
| *** flwang1 has quit IRC | 18:31 | |
| *** sidx64_ has quit IRC | 19:27 | |
| *** gsimondon has joined #openstack-containers | 19:35 | |
| *** openstackgerrit has joined #openstack-containers | 19:52 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: WIP: k8s_fedora_atomic minion upgrade https://review.openstack.org/514960 | 19:52 |
| *** pcaruana has quit IRC | 20:31 | |
| *** pcaruana has joined #openstack-containers | 20:33 | |
| *** pcaruana has quit IRC | 20:36 | |
| *** pcaruana has joined #openstack-containers | 20:39 | |
| *** pcaruana has quit IRC | 20:47 | |
| brtknr | Meeting tonight? | 20:54 |
| strigazi | +1 | 20:54 |
| *** ttsiouts has joined #openstack-containers | 20:55 | |
| strigazi | dioguerra flwang and others, Can you summarize the disappearence of master with the CA in storyboard and how to reproduce? I could not reproduce it | 21:02 |
| ttsiouts | strigazi: meeting now? | 21:03 |
| strigazi | flwang: meeting? | 21:03 |
| brtknr | ttsiouts: strigazi: yep | 21:03 |
| strigazi | I'll start it | 21:04 |
| strigazi | #startmeeting containers | 21:04 |
| openstack | Meeting started Tue Apr 9 21:04:18 2019 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:04 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:04 |
| *** openstack changes topic to " (Meeting topic: containers)" | 21:04 | |
| openstack | The meeting name has been set to 'containers' | 21:04 |
| strigazi | #topic Roll Call | 21:04 |
| *** openstack changes topic to "Roll Call (Meeting topic: containers)" | 21:04 | |
| strigazi | o/ | 21:04 |
| colin- | hello | 21:04 |
| ttsiouts | o/ | 21:04 |
| *** imdigitaljim has joined #openstack-containers | 21:06 | |
| strigazi | #topic Stories/Tasks | 21:06 |
| imdigitaljim | o/ | 21:06 |
| *** openstack changes topic to "Stories/Tasks (Meeting topic: containers)" | 21:06 | |
| brtknr | o/ | 21:06 |
| strigazi | Last week I attempted to upgrade the default version of k8s to 1.14.0 but calico v2 wasn't passing | 21:06 |
| strigazi | wasn't passing the conformance test | 21:07 |
| strigazi | I have the patch and results here: | 21:07 |
| strigazi | https://review.openstack.org/#/c/649609/ | 21:07 |
| strigazi | flwang: suggest that the latest calico, may work. I'll give it a go | 21:08 |
| imdigitaljim | we use the latest calico | 21:08 |
| colby_ | Hey Guys. Whats the latest version of kubernetes I can use on queens version of magnum (6.3.0). I tried with kube_tag=1.11.1-5 and 1.12 and both failed to build. The default 1.9.3 builds fine. | 21:08 |
| strigazi | imdigitaljim: i know, that is why I'm not asking :) | 21:08 |
| imdigitaljim | ah :D | 21:08 |
| colby_ | I mean kube_tag=v1.11.1-5 | 21:09 |
| imdigitaljim | conformance was passing as well | 21:09 |
| imdigitaljim | so you might be right | 21:09 |
| *** schaney has joined #openstack-containers | 21:10 | |
| strigazi | For upgrades, I did some modifications for the worker nodes and with the heat API works pretty well for worker and it validates the passed nodegroup. | 21:10 |
| strigazi | Some more clean up and it will works with the API. | 21:10 |
| imdigitaljim | strigazi: https://kubernetes.io/docs/setup/version-skew-policy/ | 21:10 |
| imdigitaljim | have you seen that for upgrades? | 21:11 |
| imdigitaljim | specifically https://kubernetes.io/docs/setup/version-skew-policy/#supported-component-upgrade-order | 21:11 |
| flwang | o/ | 21:11 |
| strigazi | The only missing part is the container registry on clusters | 21:11 |
| strigazi | imdigitaljim: yes, but it doesn't enforce it | 21:12 |
| flwang | sorry i'm late, NZ just had a daylight saving | 21:12 |
| strigazi | this madness with daylight will end soon, at least in the EU | 21:13 |
| flwang | strigazi: yep | 21:13 |
| flwang | strigazi: so are you still going to do the master upgrade in your existing patch? | 21:13 |
| strigazi | yes | 21:13 |
| flwang | or you will propose another one? | 21:14 |
| strigazi | this one | 21:14 |
| strigazi | flwang: do you want to the 1.14.0, it is calico related | 21:14 |
| strigazi | flwang: do you want to the 1.14.0 pathc, it is calico related | 21:14 |
| strigazi | also 1.14.1 is out | 21:14 |
| flwang | want to (do)? | 21:15 |
| strigazi | flwang: do you want to take the 1.14.0 patch, it is calico related | 21:15 |
| flwang | hehe, sure i can | 21:16 |
| flwang | but i'm busy on the auto scaling regression issue and the upgrade testing/review, is the v1.14.0 urgent for you? | 21:16 |
| strigazi | not really really urgent | 21:17 |
| flwang | strigazi: ok, then i can take it, no problem | 21:17 |
| strigazi | i said not :) | 21:17 |
| strigazi | regarding the *possible* regression with the autoscale. I wasn't able to reproduce. Can you describe it in storyboard? | 21:18 |
| flwang | strigazi: sure, are you using devstack or stable/rocky? | 21:18 |
| strigazi | devstack | 21:18 |
| flwang | and are you using the image from opentstackmagnum? | 21:19 |
| strigazi | but a in a good machine :) | 21:19 |
| strigazi | yes | 21:19 |
| flwang | are you using my patch or a home-made autoscaler yaml? | 21:19 |
| strigazi | from the CA repo, not your patch | 21:19 |
| strigazi | I don't think this is the issue https://github.com/kubernetes/autoscaler/issues/1870 | 21:20 |
| flwang | my code also from ca repo but i'd like to understand the difference, and i think it is a corner case, but we need to figure it out | 21:20 |
| flwang | strigazi: not sure, and I also got a scale down issue which autoscaler and magnum/heat are using different format of UUID | 21:21 |
| strigazi | ok, with your patch is it 100% reproducible? | 21:21 |
| flwang | i think it's reproduceible, but i don't think it's 100%, better give it a try by yourself | 21:21 |
| flwang | and that would be really appreciated | 21:22 |
| strigazi | ok, where do you test? dsvm? | 21:22 |
| strigazi | master branch? | 21:22 |
| flwang | master branch | 21:22 |
| strigazi | ok | 21:22 |
| flwang | with all latest code, including the patch NG-5 | 21:22 |
| strigazi | ok | 21:23 |
| flwang | i will dig it today as well | 21:23 |
| flwang | back to your upgrade patch, did you see all my comments? | 21:23 |
| strigazi | cool, I'll check gerrit tmr | 21:23 |
| flwang | now i can the minion upgrade works with those changed i mentioned in the patch, but in my testing, the master node will be rebuilt though i didn't change the image | 21:24 |
| strigazi | I am lost in the comments, they are too many. what changes? | 21:25 |
| strigazi | for the additional mounts it is fixed. | 21:26 |
| flwang | i suggest you review all my comments, because that took me a lot of time for testing | 21:26 |
| flwang | the additional mounts is for the minion side | 21:26 |
| flwang | i'm talking about the master | 21:26 |
| strigazi | sure, I'll address them | 21:26 |
| flwang | so do you mean i shouldn't care about the master behaviour now since you haven't done it? | 21:27 |
| strigazi | master is expected to fail atm. | 21:27 |
| flwang | strigazi: it's not "fail", it's being rebuilt | 21:27 |
| flwang | after rebuilt, master is using the new version of k8s | 21:28 |
| strigazi | that is kind of a failure :) | 21:28 |
| strigazi | I'll fix it | 21:28 |
| flwang | let me explain a bit | 21:28 |
| strigazi | I know the issue, it is because of user data | 21:29 |
| flwang | after rebuilt, all components except kubelet will be back soon, and i have to restart kubelet to get it back | 21:29 |
| flwang | it's really like the issue we're seeting for autoscaler's master rebuilt | 21:29 |
| flwang | s/seeting/seeing | 21:29 |
| strigazi | yeap, it is the issue we had with cluster_update some months ago and it was fixed | 21:30 |
| flwang | i just wanna highlight that to see if you have any idea | 21:30 |
| strigazi | yeap, it is the same issue we had with cluster_update some months ago and we fixed it | 21:30 |
| flwang | which patch fixed it? | 21:30 |
| flwang | with the autoscaler testing, i'm using master | 21:31 |
| flwang | and i also rebased the upgrade patch locally for testing | 21:31 |
| flwang | so i'm wondering which patch you're talking about | 21:31 |
| strigazi | no, I mean the cause is the same as in cluster_update in the past. | 21:31 |
| flwang | strigazi: so you mean you fixed it in your existing patch? | 21:32 |
| strigazi | https://github.com/openstack/magnum/commit/3f773f1fd045a507c3962ae509fcd57352cdc9ae | 21:32 |
| strigazi | no | 21:32 |
| strigazi | flwang: let's take a step back. | 21:32 |
| strigazi | The current patch for upgrades it is expected to "fail" for master. | 21:32 |
| strigazi | The reason is "change of user_data of the vm" | 21:33 |
| flwang | i get that | 21:33 |
| strigazi | This reason used to break cluster_update and it was fixed. | 21:33 |
| strigazi | I don't know what breaks autoscale, I''ll check. | 21:33 |
| flwang | The reason is "change of user_data of the vm" --- we have to do same thing for master like we did for minion? | 21:34 |
| flwang | to put those scripts "into" heat-container-agent? | 21:34 |
| flwang | ok, i understand | 21:34 |
| strigazi | in the current patch in gerrit I'll push a fix for master upgrade. | 21:34 |
| strigazi | yes | 21:34 |
| flwang | strigazi: cool | 21:34 |
| flwang | based on my understanding of heat update, the master nodes being rebuilt is still caused by something changed for the master | 21:36 |
| strigazi | correct | 21:36 |
| flwang | we just need to figure out what has been changed which slip out our eyes | 21:36 |
| flwang | cool, good to know we're on the same page | 21:36 |
| strigazi | for upgrades yes, for the autoscaler to be checked. | 21:36 |
| flwang | dioguerra was doubting the new security group rules from master to nodes | 21:37 |
| flwang | but still failed after revert that one | 21:37 |
| flwang | dioguerra: can you give us more details? | 21:38 |
| strigazi | it comes from Ricardo too and I also mentioned it, but I didn't have enough courage to insist | 21:38 |
| strigazi | in the patches in gerrit I mentioned that it breaks the pattern we use for ingress | 21:38 |
| strigazi | this is the remove of pors 80/443 | 21:39 |
| strigazi | the other port is ssh which change the default behaviour. | 21:39 |
| strigazi | the other port is ssh which changed the default behaviour. | 21:39 |
| strigazi | I mentioned this in gerrit as well. | 21:40 |
| flwang | if we confirmed the issue is caused by the security rules, i think we can revisit this part | 21:40 |
| strigazi | as I mentione before, in clouds that don't have octavia (like ours) or even if they do, but users don't want to use it. | 21:41 |
| strigazi | ingress works with a traefik or nginx or appscode/voyager | 21:41 |
| flwang | strigazi: i can see your point, we maybe able to introduce a config to let cloud provider to config those rules? | 21:42 |
| strigazi | using ports 80/443 in the workers | 21:42 |
| strigazi | For this, we can open when open when traefik or nginx is used or with another label | 21:43 |
| strigazi | same for ssh | 21:43 |
| flwang | strigazi: yep, that would be better and easier | 21:43 |
| strigazi | can be cloud wide or with labels | 21:44 |
| strigazi | can be cloud/magnum-deployment wide or with labels | 21:44 |
| flwang | yep, we can discuss this later for more details | 21:45 |
| strigazi | we can put additional details in storyboard | 21:46 |
| flwang | sure | 21:46 |
| brtknr | what is the update on CRUD for nodegroups ttsiouts? | 21:47 |
| flwang | strigazi: ttsiouts: is the ng-6 the last one we need for NG? on server side | 21:49 |
| strigazi | is there an NG-6 in gerrit? | 21:49 |
| flwang | https://review.openstack.org/#/c/647792/ | 21:50 |
| strigazi | before the new driver, i think this is the last one | 21:50 |
| flwang | ok, good | 21:51 |
| strigazi | and client | 21:51 |
| flwang | i'm reviewing the client now | 21:51 |
| ttsiouts | brtknr: I am refactoring the scripts for the deployment of the cluster | 21:51 |
| strigazi | I guess we need a microversion | 21:52 |
| flwang | strigazi: do you think we can start to merge upgrade api now? | 21:52 |
| ttsiouts | brtknr: in heat side | 21:52 |
| strigazi | flwang: we just need a check for master VS worker and we are good | 21:52 |
| flwang | strigazi: ok, cool, i have done the api, ref, and client, and it generally works with your functional patch | 21:53 |
| strigazi | yeap | 21:54 |
| flwang | so as long as your master upgrade work submitted, we can start to do integration testing and get things done | 21:54 |
| flwang | strigazi: thank you for working on this, i know it's a hard one | 21:54 |
| brtknr | ttsiouts: sounds good! let me know when its ready for testing :) | 21:54 |
| strigazi | :) | 21:55 |
| strigazi | just before closing, I want to make a shameless plug | 21:55 |
| strigazi | if you use barbican, you may like this one: | 21:56 |
| strigazi | https://techblog.web.cern.ch/techblog/post/helm-barbican-plugin/ | 21:56 |
| flwang | install barbican on k8s? | 21:56 |
| strigazi | Ricardo wrote an excellent plugin | 21:56 |
| strigazi | it can be easily added as a kubectl plugin | 21:57 |
| flwang | strigazi: ah, that's nice, so you still need to have barbican deployed already, right? | 21:57 |
| strigazi | yes, you need the barbican API | 21:57 |
| flwang | and then just use barbican as the secret backend for k8s? | 21:57 |
| brtknr | strigazi: I like that Kustomize is mentioned :) | 21:58 |
| flwang | that's cool, actually, we already have customer asking that | 21:58 |
| strigazi | this plugin is for client side usage. | 21:58 |
| strigazi | For KMS there is an implementation in the CPO repo | 21:58 |
| flwang | strigazi: cool | 21:59 |
| strigazi | let's end the meeting? | 22:01 |
| strigazi | Said once | 22:02 |
| strigazi | said twice | 22:02 |
| flwang | i'm good | 22:02 |
| strigazi | thanks for joining everyone | 22:03 |
| flwang | thanks strigazi | 22:03 |
| strigazi | flwang: cheers | 22:03 |
| strigazi | #endmeeting | 22:03 |
| *** openstack changes topic to "OpenStack Containers Team" | 22:03 | |
| openstack | Meeting ended Tue Apr 9 22:03:11 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:03 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/containers/2019/containers.2019-04-09-21.04.html | 22:03 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/containers/2019/containers.2019-04-09-21.04.txt | 22:03 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/containers/2019/containers.2019-04-09-21.04.log.html | 22:03 |
| flwang | strigazi: any ETA about the master upgrade work? | 22:03 |
| strigazi | tomorrow if nothing bad happens | 22:03 |
| flwang | that's fantastic | 22:04 |
| flwang | I love your efficiency when you're free ;) | 22:04 |
| strigazi | :) | 22:05 |
| flwang | thank you and have a good night | 22:05 |
| strigazi | thanks, have a nic day! | 22:05 |
| openstackgerrit | Ricardo Rocha proposed openstack/magnum master: [k8s] Add nginx based ingress controller https://review.openstack.org/648655 | 22:06 |
| brtknr | strigazi: are you still there? | 22:12 |
| brtknr | would you mind replying to my questions on the multi-nic thread on gerrit when you get a chance? | 22:12 |
| *** hongbin has joined #openstack-containers | 22:13 | |
| brtknr | i am not sure what you mean by you cant access logs when using floating ip... | 22:13 |
| *** ttsiouts has quit IRC | 22:16 | |
| *** ttsiouts has joined #openstack-containers | 22:17 | |
| *** ttsiouts has quit IRC | 22:21 | |
| *** rcernin has joined #openstack-containers | 22:28 | |
| colby_ | Does anyone know the latest kubernetes version that works with queens magnum (6.3.0)? The default 1.9.3 builds fine, but I tried 1.11.1-5 and 1.12 and both fail to build, with kubernetes never starting on the master (fedora-atomic-27) | 23:00 |
| *** livelace has quit IRC | 23:14 | |
| flwang | colby_: better let us know the error | 23:26 |
| flwang | colby_: i think we tested 1.9.3 for queens, and for rocky, it has been upgrade to 1.11.x | 23:26 |
| colby_ | ah I think I see. there is no 1.11.1-5 or 1.12 image. There is 1.11.5-1 and 1.12.5 | 23:27 |
| colby_ | you would think that atomic install would give an error is the image does not exist instead of just no response | 23:27 |
| flwang | colby_: you should be able to see the log from /var/log/cloud-init-output.log for sure | 23:30 |
| colby_ | yea it was just empty after the atomic install. I tried running the command from command line and it just silently exits | 23:31 |
| flwang | check the images we have on openstackmagnum dockerhub | 23:31 |
| colby_ | yea I did that...thats how I noticed they did not exist :) | 23:32 |
| colby_ | I switched the 1 and 5 in 1.11.5-1 on accident | 23:32 |
| *** hongbin has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!