*** rcernin has joined #openstack-containers | 01:06 | |
*** xinliang has joined #openstack-containers | 01:23 | |
*** xinliang has quit IRC | 01:29 | |
*** xinliang has joined #openstack-containers | 01:30 | |
*** xinliang has quit IRC | 01:37 | |
*** vesper has joined #openstack-containers | 01:38 | |
*** vesper11 has quit IRC | 01:39 | |
*** vishalmanchanda has joined #openstack-containers | 02:10 | |
openstackgerrit | Feilong Wang proposed openstack/magnum master: [WIP][k8s]Upgrade calico/coredns to the latest stable version https://review.opendev.org/705599 | 03:10 |
---|---|---|
openstackgerrit | Feilong Wang proposed openstack/magnum master: [WIP][k8s]Upgrade calico/coredns to the latest stable version https://review.opendev.org/705599 | 03:13 |
*** ykarel|away is now known as ykarel | 03:30 | |
*** rcernin has quit IRC | 03:31 | |
*** ramishra has joined #openstack-containers | 03:50 | |
*** ramishra has quit IRC | 04:01 | |
*** ramishra has joined #openstack-containers | 04:24 | |
*** iokiwi has quit IRC | 04:45 | |
*** udesale has joined #openstack-containers | 04:53 | |
*** hongbin has joined #openstack-containers | 04:55 | |
*** rcernin has joined #openstack-containers | 04:57 | |
*** rcernin has quit IRC | 04:58 | |
*** rcernin has joined #openstack-containers | 04:58 | |
*** hongbin has quit IRC | 05:28 | |
*** rcernin has quit IRC | 06:03 | |
*** ykarel is now known as ykarel|lunch | 07:42 | |
brtknr | flwang: hi, just wondering if metric server works for you? Mine keeps doing a crashloopback | 07:56 |
*** elenalindq has joined #openstack-containers | 08:16 | |
*** rcernin has joined #openstack-containers | 08:32 | |
*** flwang1 has joined #openstack-containers | 08:35 | |
*** rcernin has quit IRC | 08:48 | |
*** ykarel|lunch is now known as ykarel | 08:52 | |
*** rcernin has joined #openstack-containers | 08:54 | |
*** xinliang has joined #openstack-containers | 08:58 | |
strigazi | o/ | 08:59 |
flwang1 | strigazi: hey | 08:59 |
strigazi | meeting? no? | 09:00 |
strigazi | brtknr yes it works | 09:01 |
flwang1 | #startmeeting magnum | 09:01 |
openstack | Meeting started Wed Feb 5 09:01:39 2020 UTC and is due to finish in 60 minutes. The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:01 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:01 |
*** openstack changes topic to " (Meeting topic: magnum)" | 09:01 | |
openstack | The meeting name has been set to 'magnum' | 09:01 |
flwang1 | #topic roll call | 09:01 |
*** openstack changes topic to "roll call (Meeting topic: magnum)" | 09:01 | |
strigazi | o/ | 09:01 |
flwang1 | o/ | 09:01 |
flwang1 | brtknr: ? | 09:02 |
*** xinliang has quit IRC | 09:02 | |
flwang1 | strigazi: so far i think just you and me | 09:02 |
flwang1 | strigazi: is there anything you want to discuss? | 09:03 |
brtknr | flwang1: hi | 09:03 |
brtknr | o/ | 09:03 |
brtknr | was scoffing down breakfast, apologies | 09:04 |
flwang1 | brtknr: all good | 09:04 |
flwang1 | brtknr: strigazi: do you guys have special topic to discuss? | 09:04 |
flwang1 | otherwise we go through the agenda? | 09:04 |
strigazi | flwang1: brtknr: the new heat agent logs are worse | 09:04 |
brtknr | elaborate | 09:05 |
strigazi | flwang1: brtknr: the output of a command must be printed when it is executed | 09:05 |
brtknr | +1 | 09:05 |
strigazi | flwang1: brtknr: atm everything is printed at the end | 09:06 |
brtknr | there is some time where nothing gets printed | 09:06 |
brtknr | would be nice to fix that | 09:06 |
strigazi | flwang1: brtknr: can't catch trhings that are stucj | 09:06 |
flwang1 | strigazi: yep, i notice that as well | 09:06 |
strigazi | flwang1: brtknr: can't catch things that are stuck | 09:06 |
strigazi | flwang1: brtknr: when did this happen? | 09:06 |
flwang1 | i'd like to see the logs being printed for each execution | 09:06 |
flwang1 | i think since last time brtknr fixed it? | 09:07 |
brtknr | its been like this since we moved everything from cloud-init-output to journalctl | 09:07 |
brtknr | when i fixed it, nothing was getting printed at all | 09:07 |
*** rcernin has quit IRC | 09:07 | |
strigazi | "nothing was getting printed at all" when was this true? | 09:07 |
brtknr | when i fixed it, everything was getting printed on a single line | 09:07 |
strigazi | ok | 09:08 |
brtknr | i simply changed it so that newlines were parsed correctly | 09:08 |
strigazi | I don't remember that. Anything we can do? | 09:08 |
flwang1 | nothing special, just review the code and see how can we fix it | 09:09 |
strigazi | is it our code or os-collect-config? | 09:09 |
flwang1 | probably os-collect-config, i need to check the code, can't remember the details | 09:10 |
brtknr | ok who is going to look into this? | 09:13 |
flwang1 | i just created a story to track this https://storyboard.openstack.org/#!/story/2007256 | 09:14 |
flwang1 | strigazi: can you work on this? | 09:14 |
strigazi | flwang1: I can have a look | 09:15 |
brtknr | next topic? | 09:16 |
strigazi | where is the etherpad? | 09:17 |
flwang1 | https://etherpad.openstack.org/p/magnum-weekly-meeting | 09:17 |
strigazi | it is not in he channel anymore | 09:17 |
strigazi | thx | 09:17 |
flwang1 | shall we go through the agenda? | 09:20 |
brtknr | yes | 09:20 |
brtknr | are you guys happy to provide feedback for cinder_csi_enabled | 09:20 |
strigazi | is it ready? | 09:21 |
brtknr | to support out of tree cinder? | 09:21 |
brtknr | yep but i dont understand waht the pep8 failure is all about | 09:21 |
flwang1 | brtknr: so you have already tested and it works? | 09:21 |
flwang1 | i'm happy to test it | 09:21 |
brtknr | yep ive tested and it works | 09:22 |
flwang1 | great | 09:22 |
brtknr | works for xinliang too | 09:22 |
flwang1 | i will help take a look the pep8 and docs job failure | 09:23 |
strigazi | brtknr: I will have a look, at CERN we have two more CSI drivers. Need to check that this is compatible. | 09:23 |
brtknr | there are some generic csi components, only one of them is cinder_csi replated | 09:25 |
brtknr | related | 09:25 |
flwang1 | strigazi: if cern can give some comments on this csi patch, it would be great | 09:25 |
strigazi | brtknr: don't you need any of: --feature-gates=CSINodeInfo=true,CSIDriverRegistry=true ? | 09:25 |
strigazi | brtknr: or --runtime-config=storage.k8s.io/v1alpha1=true" ? | 09:25 |
brtknr | didnt need those | 09:26 |
brtknr | seemed to work with v1.16.x and 1.17.x | 09:26 |
strigazi | interesting | 09:26 |
*** rcernin has joined #openstack-containers | 09:27 | |
strigazi | ok, I will test | 09:27 |
brtknr | CSIDriverRegistry went into beta from 1.14 | 09:27 |
brtknr | same for CSINodeInfo | 09:27 |
brtknr | so true by default | 09:27 |
strigazi | brtknr: means on by default? | 09:27 |
strigazi | ok | 09:27 |
strigazi | The world will end based on a default | 09:28 |
brtknr | XD | 09:28 |
brtknr | I'll quote you on that when the world ends | 09:28 |
strigazi | 100% sure | 09:29 |
strigazi | some will change the nuke_everything default to true in qa, and this it | 09:29 |
strigazi | anyway, let's stay on track, I will test it | 09:30 |
strigazi | brtknr: would you like to do manila provisioner too? | 09:30 |
strigazi | I'm good, let's move on? | 09:31 |
flwang1 | ok | 09:32 |
brtknr | strigazi: i tried the manila provisioner but lost my patience with it | 09:32 |
flwang1 | brtknr: what's the nginx issue? | 09:32 |
*** rcernin has quit IRC | 09:33 | |
brtknr | with regards to nginx and traefik, at the moment, we need to label an ingress node. we wanted to explore adding another option which runs a deployment with n replicas behind a load balancer | 09:35 |
strigazi | brtknr: what do you propose? | 09:36 |
brtknr | effectively a label which changes nginx and traefik services from ClusterIP to LoadBalancer type | 09:37 |
brtknr | fairly small change | 09:37 |
strigazi | and DS to deploy and hostNetwork: false | 09:37 |
brtknr | it can be ClusterIP by default | 09:37 |
strigazi | and remove node-selector | 09:37 |
brtknr | possibly those too :S needs to look into it more closely | 09:39 |
strigazi | as mentioned before, just don't break the default :) | 09:40 |
brtknr | ok im glad you're okay to support the alternative, ill refine what it would involve a bit more in a PS | 09:41 |
strigazi | for prometheus, ping dioguerra , I think it works but double check with him | 09:41 |
flwang1 | re prometheus, what's the status of removing heapster and adding metrics-server | 09:42 |
brtknr | i take back the promethus issue | 09:42 |
strigazi | it is done | 09:42 |
flwang1 | i can see the metrics-server pods is broken on my local | 09:42 |
brtknr | metrics server doesnt work for me either at 9.2.0 | 09:42 |
flwang1 | strigazi: you mean the heapster->metrics-server work is done, right | 09:43 |
flwang1 | ? | 09:43 |
brtknr | lemme grab the log | 09:43 |
strigazi | yes | 09:43 |
strigazi | metrics server works for me. Only the logs of the master node are not collected due to the cert | 09:43 |
flwang1 | strigazi: ok, i will test again | 09:44 |
flwang1 | what's the k8s version you're using? | 09:44 |
strigazi | 17.2 | 09:44 |
flwang1 | fairly new :) | 09:45 |
flwang1 | ok, i will give it a try | 09:45 |
flwang1 | let's move on? | 09:45 |
strigazi | yes | 09:46 |
flwang1 | https://review.opendev.org/#/c/700565 work for ARM | 09:46 |
flwang1 | personally i'm ok with that | 09:46 |
brtknr | i created a story for the metrics-server issue: https://storyboard.openstack.org/#!/story/2007264 | 09:46 |
brtknr | do you need special label for metrics server? | 09:47 |
brtknr | flwang1: have you tested https://storyboard.openstack.org/#!/story/2007264 | 09:47 |
brtknr | flwang1: have you tested https://review.opendev.org/#/c/700565 | 09:47 |
strigazi | https://storyboard.openstack.org/#!/story/2007264 this is the heat-agent | 09:47 |
brtknr | i created a story for the metrics-server issue: https://storyboard.openstack.org/#!/story/2007265 | 09:48 |
brtknr | sorry | 09:48 |
flwang1 | no i haven't, i just had a quick look about the code | 09:48 |
brtknr | the ARCH change looks trivial, i wanted to get your thoughts on the general approach | 09:49 |
strigazi | brtknr: DNS k8s-podman-calico-k7tz6itxx7k6-node-0 on 10.254.0.10:53: no such host | 09:49 |
flwang1 | brtknr: will do | 09:49 |
strigazi | for ARCH I will that the defailt x86_64 works | 09:49 |
brtknr | i.e. determining arch inside /write-heat-params.sh script | 09:50 |
strigazi | I can't test arm | 09:50 |
brtknr | strigazi: what is the fix for DNS k8s-podman-calico-k7tz6itxx7k6-node-0 on 10.254.0.10:53: no such host | 09:51 |
strigazi | left a comment | 09:51 |
brtknr | strigazi: ok thanks, that was not obvious to me :) | 09:53 |
strigazi | https://pbs.twimg.com/media/C9o9ZO0UwAAi3ib.jpg | 09:53 |
brtknr | lol | 09:54 |
strigazi | defaults usually assume DNS. I have more scenarios for the end of the world, all of them include DNS as well | 09:54 |
flwang1 | move on? | 09:55 |
flwang1 | we have only 5 mins | 09:55 |
jakeyip | hi all | 09:56 |
flwang1 | brtknr: what's this 'Modify default-worker flavor after cluster creation'? | 09:56 |
flwang1 | jakeyip: hi | 09:56 |
brtknr | flwang1: so we have a use case where we want to be able to change default worker flavor after the cluster is already creted | 09:58 |
brtknr | at the moment, we can delete nodegroup and create a new nodegroup with different flavor | 09:58 |
brtknr | but we cant do this for default worker | 09:58 |
brtknr | i was wondering how involved a change it would be to allow update of default-worker flavor | 09:59 |
strigazi | I think the best approach is to make the default worker deletable | 09:59 |
flwang1 | brtknr: maybe we should allow create an empty cluster | 10:00 |
flwang1 | like EKS | 10:00 |
strigazi | changing the flavor sounds orthogonal to nova to me | 10:00 |
flwang1 | i don't think resizing the instance is a good idea | 10:00 |
strigazi | +1 | 10:00 |
brtknr | yes i'd prefer to make the default worker deletable | 10:01 |
strigazi | EKS logic gives little value | 10:01 |
brtknr | and create cluster with 0 workers | 10:01 |
flwang1 | btw, i'd like to propose a patch to support distribute master nodes to different AZs, strigazi, do you like the idea? | 10:02 |
brtknr | flwang1: is that what you mean by empty cluster ? | 10:02 |
flwang1 | brtknr: no, different idea | 10:02 |
brtknr | or does this also include 0 masters? | 10:02 |
jakeyip | flwang1: +1 we will like that for workers | 10:02 |
flwang1 | it's like the Regional cluster in GKE | 10:02 |
flwang1 | jakeyip: for workers, we already can support it with node groups, but i know where you come from | 10:03 |
strigazi | master across AZs could work, but it feels hacky | 10:04 |
strigazi | I mean you can deploy N NGs for masters by default | 10:04 |
strigazi | this sounds better | 10:04 |
flwang1 | strigazi: what do you mean deploy N NGs for masters? | 10:05 |
strigazi | if you have 3 AZs in your cloud, deploy three master NGs | 10:05 |
strigazi | on creation | 10:05 |
strigazi | N=3 in this case | 10:06 |
brtknr | that sounds like a better approach | 10:06 |
flwang1 | you mean create 3NGs for master nodes? | 10:06 |
flwang1 | how can i do that? | 10:06 |
strigazi | yes | 10:06 |
strigazi | one sec | 10:07 |
strigazi | still looking | 10:10 |
strigazi | I don't remember were it is | 10:10 |
brtknr | i didnt know you could create master N G | 10:11 |
brtknr | i didnt know you could create master NG | 10:11 |
flwang1 | yep, i thought master can only be in default-master NG | 10:11 |
strigazi | you can't atm, we have similar code though | 10:11 |
flwang1 | that's why i asked | 10:12 |
flwang1 | strigazi: can you please share the code with me when you found it? | 10:12 |
flwang1 | i'm keen to learn that | 10:12 |
strigazi | found ir | 10:13 |
strigazi | found it | 10:13 |
strigazi | wait | 10:13 |
strigazi | https://github.com/openstack/magnum/blob/e52f77b299c50f004ee5a875c68d7f129b88a2af/magnum/conductor/handlers/cluster_conductor.py#L56 | 10:14 |
strigazi | you can add the additional NGs there, needs some extra work for the LB to be shared across the stacks | 10:14 |
flwang1 | hmm... what's the benefit to put masters into different NG? | 10:15 |
*** ramishra has quit IRC | 10:15 | |
flwang1 | anyway, thanks, i will think about this | 10:15 |
strigazi | to spawn across AZs? | 10:15 |
strigazi | that is the goal no? | 10:15 |
strigazi | Then you will have a NG with nodes in many AZs inside? | 10:16 |
flwang1 | if there is an AZ list, we should be able to spawn masters into different AZ | 10:16 |
strigazi | So then all the nodegroup work we did is kind of worthless | 10:16 |
strigazi | as you want | 10:17 |
flwang1 | strigazi: please don't say words like this | 10:17 |
flwang1 | i'm not trying to make anyone's work worthless | 10:17 |
strigazi | The biggest use case of NGs was AZs | 10:18 |
*** xinliang has joined #openstack-containers | 10:18 | |
flwang1 | ok, then if you guys do have a good solution for this, please share, otherwise, what i am doing is trying to figure out a way to achieve that | 10:19 |
strigazi | well, I might be wrong. IN GKE they spawn AZs across zones, if I read correctly | 10:20 |
strigazi | https://cloud.google.com/kubernetes-engine/docs/concepts/node-pools | 10:20 |
strigazi | https://cloud.google.com/kubernetes-engine/docs/concepts/node-pools#nodes_in_multi-zonal_clusters | 10:20 |
*** ramishra has joined #openstack-containers | 10:21 | |
strigazi | Maybe the thing with the list works | 10:22 |
flwang1 | https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters | 10:22 |
strigazi | It will require some serious hacking in the heat templates, maybe | 10:22 |
flwang1 | let's use a spec to track this work | 10:22 |
flwang1 | i will start to draft that | 10:22 |
flwang1 | brtknr: strigazi: could you please help review this one Volume AZ regression fix - https://review.opendev.org/705592 ? | 10:23 |
flwang1 | it's kind of a regression issue, i didn't fix the volume AZ issue at the first shot | 10:23 |
strigazi | let's wrap the meeting? with train many bugs arrived | 10:23 |
flwang1 | cinder doesn't respect "" as AZ | 10:24 |
flwang1 | strigazi: sure | 10:24 |
strigazi | flwang1: I will have a look, I might have time only on monday though | 10:24 |
flwang1 | #endmeeting | 10:24 |
*** openstack changes topic to "OpenStack Containers Team | Meeting: every Wednesday @ 9AM UTC | Agenda: https://etherpad.openstack.org/p/magnum-weekly-meeting" | 10:24 | |
openstack | Meeting ended Wed Feb 5 10:24:23 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 10:24 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-02-05-09.01.html | 10:24 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-02-05-09.01.txt | 10:24 |
openstack | Log: http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-02-05-09.01.log.html | 10:24 |
flwang1 | strigazi: thanks, it would be really appreciated | 10:24 |
flwang1 | share on me, i didn't get it fully tested | 10:24 |
jakeyip | the volume az patch is similar to one of our custom patch | 10:24 |
flwang1 | jakeyip: thanks, if you can, please help review it | 10:25 |
jakeyip | not sure if it's totally correct though... that's why I didn't submit it upstream | 10:25 |
jakeyip | cinder AZ may not be equal to nova AZ | 10:25 |
flwang1 | jakeyip: we have to use the condition, since cinder is not smart enough | 10:25 |
* strigazi goes bug hunting | 10:26 | |
flwang1 | jakeyip: yep, i know, it's a good point. | 10:26 |
jakeyip | I'm referring more to the original patch rather than the regression. I agree with the regression fix. The logic for the original patch may not be correct globally | 10:26 |
flwang1 | jakeyip: but most of the companies will design it to make sure they're same to avoid troubles | 10:26 |
jakeyip | It is OK for us so we carry such a patch in house ( -volume-type +availability-zone) | 10:27 |
jakeyip | it might not be OK for the world | 10:27 |
jakeyip | what is needed might be a way to override Cinder AZ | 10:27 |
flwang1 | jakeyip: it would be a bit overkill if we introduce another label for cinder az :( | 10:27 |
jakeyip | take it from the AZ parameter but allow override if possible | 10:28 |
jakeyip | yeap. hence also the note I put on agenda | 10:28 |
flwang1 | i'm going to introduce an AZ list as config option | 10:28 |
jakeyip | just suggestions don't have to implement until someone files a bug report :P | 10:28 |
flwang1 | if it's set, magnum will randomly pick one from the list | 10:28 |
jakeyip | our patch - https://github.com/NeCTAR-RC/magnum/commit/e3e4afe10bf22aee7824dfc0df5d30b2c55502e8 | 10:29 |
flwang1 | jakeyip: labels are an old topic and we don't have a perfect answer yet | 10:29 |
flwang1 | jakeyip: that's a lazy patch :) | 10:30 |
jakeyip | that's why it's not upstream :P | 10:30 |
flwang1 | jakeyip: i have to go | 10:30 |
jakeyip | seeya | 10:30 |
flwang1 | jakeyip: please help review the patch, we need you guys input | 10:31 |
jakeyip | ok | 10:31 |
flwang1 | thanks | 10:32 |
flwang1 | ttyl | 10:32 |
openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: More verbose logs for cluster ops https://review.opendev.org/705922 | 10:50 |
*** udesale_ has joined #openstack-containers | 11:13 | |
*** udesale has quit IRC | 11:16 | |
*** ramishra has quit IRC | 11:18 | |
*** ramishra has joined #openstack-containers | 11:22 | |
brtknr | strigazi: im trying to work out how i can create a cluster with 0 workers | 11:23 |
*** pcaruana has quit IRC | 11:37 | |
*** pcaruana has joined #openstack-containers | 11:50 | |
openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Add cinder_csi_enabled label https://review.opendev.org/700357 | 12:05 |
*** lpetrut has joined #openstack-containers | 13:24 | |
*** ykarel is now known as ykarel|away | 13:27 | |
*** pcaruana has quit IRC | 14:17 | |
*** ykarel|away is now known as ykarel | 14:20 | |
*** ykarel is now known as ykarel|away | 15:08 | |
*** jmlowe has joined #openstack-containers | 15:13 | |
*** jmlowe has quit IRC | 15:17 | |
*** jmlowe has joined #openstack-containers | 15:17 | |
openstackgerrit | Bharat Kunwar proposed openstack/magnum master: [k8s] Fix metrics-server CrashLoopBack https://review.opendev.org/705984 | 15:25 |
*** lpetrut has quit IRC | 15:28 | |
*** ramishra has quit IRC | 15:37 | |
*** ianychoi_ is now known as ianychoi | 15:40 | |
*** osmanlicilegi has quit IRC | 15:42 | |
*** osmanlicilegi has joined #openstack-containers | 15:42 | |
*** pcaruana has joined #openstack-containers | 15:49 | |
*** openstackstatus has joined #openstack-containers | 16:04 | |
*** ChanServ sets mode: +v openstackstatus | 16:04 | |
*** jmlowe has quit IRC | 16:08 | |
*** ramishra has joined #openstack-containers | 16:16 | |
*** kklimonda has quit IRC | 16:22 | |
*** kklimonda has joined #openstack-containers | 16:22 | |
*** udesale_ has quit IRC | 16:24 | |
*** jmlowe has joined #openstack-containers | 16:47 | |
*** flwang has quit IRC | 17:48 | |
*** NobodyCam has quit IRC | 17:50 | |
*** NobodyCam has joined #openstack-containers | 17:51 | |
*** jmlowe has quit IRC | 17:54 | |
*** NobodyCam has quit IRC | 19:39 | |
*** kklimonda has quit IRC | 19:40 | |
*** jmlowe has joined #openstack-containers | 20:00 | |
*** jmlowe has quit IRC | 20:01 | |
*** jmlowe has joined #openstack-containers | 20:15 | |
*** mgariepy has quit IRC | 20:54 | |
*** jmlowe has quit IRC | 21:08 | |
*** jmlowe has joined #openstack-containers | 21:10 | |
*** jmlowe has quit IRC | 21:22 | |
*** elenalindq has quit IRC | 21:30 | |
*** yolanda has joined #openstack-containers | 22:35 | |
*** kklimonda has joined #openstack-containers | 22:59 | |
*** NobodyCam has joined #openstack-containers | 23:00 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!