09:01:39 <flwang1> #startmeeting magnum 09:01:40 <openstack> Meeting started Wed Feb 5 09:01:39 2020 UTC and is due to finish in 60 minutes. The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot. 09:01:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 09:01:43 <openstack> The meeting name has been set to 'magnum' 09:01:51 <flwang1> #topic roll call 09:01:57 <strigazi> o/ 09:01:58 <flwang1> o/ 09:02:01 <flwang1> brtknr: ? 09:02:32 <flwang1> strigazi: so far i think just you and me 09:03:38 <flwang1> strigazi: is there anything you want to discuss? 09:03:42 <brtknr> flwang1: hi 09:03:44 <brtknr> o/ 09:04:07 <brtknr> was scoffing down breakfast, apologies 09:04:16 <flwang1> brtknr: all good 09:04:35 <flwang1> brtknr: strigazi: do you guys have special topic to discuss? 09:04:46 <flwang1> otherwise we go through the agenda? 09:04:55 <strigazi> flwang1: brtknr: the new heat agent logs are worse 09:05:15 <brtknr> elaborate 09:05:42 <strigazi> flwang1: brtknr: the output of a command must be printed when it is executed 09:05:49 <brtknr> +1 09:06:06 <strigazi> flwang1: brtknr: atm everything is printed at the end 09:06:10 <brtknr> there is some time where nothing gets printed 09:06:19 <brtknr> would be nice to fix that 09:06:22 <strigazi> flwang1: brtknr: can't catch trhings that are stucj 09:06:26 <flwang1> strigazi: yep, i notice that as well 09:06:27 <strigazi> flwang1: brtknr: can't catch things that are stuck 09:06:42 <strigazi> flwang1: brtknr: when did this happen? 09:06:49 <flwang1> i'd like to see the logs being printed for each execution 09:07:08 <flwang1> i think since last time brtknr fixed it? 09:07:10 <brtknr> its been like this since we moved everything from cloud-init-output to journalctl 09:07:25 <brtknr> when i fixed it, nothing was getting printed at all 09:07:53 <strigazi> "nothing was getting printed at all" when was this true? 09:07:58 <brtknr> when i fixed it, everything was getting printed on a single line 09:08:08 <strigazi> ok 09:08:34 <brtknr> i simply changed it so that newlines were parsed correctly 09:08:38 <strigazi> I don't remember that. Anything we can do? 09:09:11 <flwang1> nothing special, just review the code and see how can we fix it 09:09:36 <strigazi> is it our code or os-collect-config? 09:10:15 <flwang1> probably os-collect-config, i need to check the code, can't remember the details 09:13:46 <brtknr> ok who is going to look into this? 09:14:38 <flwang1> i just created a story to track this https://storyboard.openstack.org/#!/story/2007256 09:14:49 <flwang1> strigazi: can you work on this? 09:15:18 <strigazi> flwang1: I can have a look 09:16:18 <brtknr> next topic? 09:17:05 <strigazi> where is the etherpad? 09:17:19 <flwang1> https://etherpad.openstack.org/p/magnum-weekly-meeting 09:17:24 <strigazi> it is not in he channel anymore 09:17:49 <strigazi> thx 09:20:19 <flwang1> shall we go through the agenda? 09:20:28 <brtknr> yes 09:20:49 <brtknr> are you guys happy to provide feedback for cinder_csi_enabled 09:21:03 <strigazi> is it ready? 09:21:07 <brtknr> to support out of tree cinder? 09:21:25 <brtknr> yep but i dont understand waht the pep8 failure is all about 09:21:51 <flwang1> brtknr: so you have already tested and it works? 09:21:55 <flwang1> i'm happy to test it 09:22:07 <brtknr> yep ive tested and it works 09:22:17 <flwang1> great 09:22:23 <brtknr> works for xinliang too 09:23:09 <flwang1> i will help take a look the pep8 and docs job failure 09:23:26 <strigazi> brtknr: I will have a look, at CERN we have two more CSI drivers. Need to check that this is compatible. 09:25:23 <brtknr> there are some generic csi components, only one of them is cinder_csi replated 09:25:26 <brtknr> related 09:25:27 <flwang1> strigazi: if cern can give some comments on this csi patch, it would be great 09:25:30 <strigazi> brtknr: don't you need any of: --feature-gates=CSINodeInfo=true,CSIDriverRegistry=true ? 09:25:45 <strigazi> brtknr: or --runtime-config=storage.k8s.io/v1alpha1=true" ? 09:26:03 <brtknr> didnt need those 09:26:27 <brtknr> seemed to work with v1.16.x and 1.17.x 09:26:46 <strigazi> interesting 09:27:19 <strigazi> ok, I will test 09:27:21 <brtknr> CSIDriverRegistry went into beta from 1.14 09:27:35 <brtknr> same for CSINodeInfo 09:27:43 <brtknr> so true by default 09:27:48 <strigazi> brtknr: means on by default? 09:27:49 <strigazi> ok 09:28:09 <strigazi> The world will end based on a default 09:28:21 <brtknr> XD 09:28:37 <brtknr> I'll quote you on that when the world ends 09:29:07 <strigazi> 100% sure 09:29:33 <strigazi> some will change the nuke_everything default to true in qa, and this it 09:30:11 <strigazi> anyway, let's stay on track, I will test it 09:30:36 <strigazi> brtknr: would you like to do manila provisioner too? 09:31:16 <strigazi> I'm good, let's move on? 09:32:38 <flwang1> ok 09:32:56 <brtknr> strigazi: i tried the manila provisioner but lost my patience with it 09:32:57 <flwang1> brtknr: what's the nginx issue? 09:35:04 <brtknr> with regards to nginx and traefik, at the moment, we need to label an ingress node. we wanted to explore adding another option which runs a deployment with n replicas behind a load balancer 09:36:25 <strigazi> brtknr: what do you propose? 09:37:10 <brtknr> effectively a label which changes nginx and traefik services from ClusterIP to LoadBalancer type 09:37:17 <brtknr> fairly small change 09:37:32 <strigazi> and DS to deploy and hostNetwork: false 09:37:35 <brtknr> it can be ClusterIP by default 09:37:47 <strigazi> and remove node-selector 09:39:02 <brtknr> possibly those too :S needs to look into it more closely 09:40:03 <strigazi> as mentioned before, just don't break the default :) 09:41:06 <brtknr> ok im glad you're okay to support the alternative, ill refine what it would involve a bit more in a PS 09:41:43 <strigazi> for prometheus, ping dioguerra , I think it works but double check with him 09:42:08 <flwang1> re prometheus, what's the status of removing heapster and adding metrics-server 09:42:13 <brtknr> i take back the promethus issue 09:42:16 <strigazi> it is done 09:42:25 <flwang1> i can see the metrics-server pods is broken on my local 09:42:44 <brtknr> metrics server doesnt work for me either at 9.2.0 09:43:00 <flwang1> strigazi: you mean the heapster->metrics-server work is done, right 09:43:02 <flwang1> ？ 09:43:04 <brtknr> lemme grab the log 09:43:05 <strigazi> yes 09:43:41 <strigazi> metrics server works for me. Only the logs of the master node are not collected due to the cert 09:44:09 <flwang1> strigazi: ok, i will test again 09:44:15 <flwang1> what's the k8s version you're using? 09:44:48 <strigazi> 17.2 09:45:26 <flwang1> fairly new :) 09:45:32 <flwang1> ok, i will give it a try 09:45:36 <flwang1> let's move on? 09:46:17 <strigazi> yes 09:46:22 <flwang1> https://review.opendev.org/#/c/700565 work for ARM 09:46:35 <flwang1> personally i'm ok with that 09:46:45 <brtknr> i created a story for the metrics-server issue: https://storyboard.openstack.org/#!/story/2007264 09:47:20 <brtknr> do you need special label for metrics server? 09:47:29 <brtknr> flwang1: have you tested https://storyboard.openstack.org/#!/story/2007264 09:47:38 <brtknr> flwang1: have you tested https://review.opendev.org/#/c/700565 09:47:56 <strigazi> https://storyboard.openstack.org/#!/story/2007264 this is the heat-agent 09:48:36 <brtknr> i created a story for the metrics-server issue: https://storyboard.openstack.org/#!/story/2007265 09:48:39 <brtknr> sorry 09:48:40 <flwang1> no i haven't, i just had a quick look about the code 09:49:20 <brtknr> the ARCH change looks trivial, i wanted to get your thoughts on the general approach 09:49:24 <strigazi> brtknr: DNS k8s-podman-calico-k7tz6itxx7k6-node-0 on 10.254.0.10:53: no such host 09:49:34 <flwang1> brtknr: will do 09:49:58 <strigazi> for ARCH I will that the defailt x86_64 works 09:50:05 <brtknr> i.e. determining arch inside /write-heat-params.sh script 09:50:10 <strigazi> I can't test arm 09:51:07 <brtknr> strigazi: what is the fix for DNS k8s-podman-calico-k7tz6itxx7k6-node-0 on 10.254.0.10:53: no such host 09:51:31 <strigazi> left a comment 09:53:11 <brtknr> strigazi: ok thanks, that was not obvious to me :) 09:53:52 <strigazi> https://pbs.twimg.com/media/C9o9ZO0UwAAi3ib.jpg 09:54:09 <brtknr> lol 09:54:54 <strigazi> defaults usually assume DNS. I have more scenarios for the end of the world, all of them include DNS as well 09:55:27 <flwang1> move on? 09:55:32 <flwang1> we have only 5 mins 09:56:07 <jakeyip> hi all 09:56:38 <flwang1> brtknr: what's this 'Modify default-worker flavor after cluster creation'? 09:56:41 <flwang1> jakeyip: hi 09:58:09 <brtknr> flwang1: so we have a use case where we want to be able to change default worker flavor after the cluster is already creted 09:58:46 <brtknr> at the moment, we can delete nodegroup and create a new nodegroup with different flavor 09:58:53 <brtknr> but we cant do this for default worker 09:59:24 <brtknr> i was wondering how involved a change it would be to allow update of default-worker flavor 09:59:55 <strigazi> I think the best approach is to make the default worker deletable 10:00:05 <flwang1> brtknr: maybe we should allow create an empty cluster 10:00:07 <flwang1> like EKS 10:00:22 <strigazi> changing the flavor sounds orthogonal to nova to me 10:00:45 <flwang1> i don't think resizing the instance is a good idea 10:00:51 <strigazi> +1 10:01:09 <brtknr> yes i'd prefer to make the default worker deletable 10:01:13 <strigazi> EKS logic gives little value 10:01:22 <brtknr> and create cluster with 0 workers 10:02:10 <flwang1> btw, i'd like to propose a patch to support distribute master nodes to different AZs, strigazi, do you like the idea? 10:02:22 <brtknr> flwang1: is that what you mean by empty cluster ? 10:02:33 <flwang1> brtknr: no, different idea 10:02:35 <brtknr> or does this also include 0 masters? 10:02:41 <jakeyip> flwang1: +1 we will like that for workers 10:02:48 <flwang1> it's like the Regional cluster in GKE 10:03:33 <flwang1> jakeyip: for workers, we already can support it with node groups, but i know where you come from 10:04:15 <strigazi> master across AZs could work, but it feels hacky 10:04:41 <strigazi> I mean you can deploy N NGs for masters by default 10:04:47 <strigazi> this sounds better 10:05:07 <flwang1> strigazi: what do you mean deploy N NGs for masters? 10:05:48 <strigazi> if you have 3 AZs in your cloud, deploy three master NGs 10:05:54 <strigazi> on creation 10:06:03 <strigazi> N=3 in this case 10:06:22 <brtknr> that sounds like a better approach 10:06:29 <flwang1> you mean create 3NGs for master nodes? 10:06:34 <flwang1> how can i do that? 10:06:58 <strigazi> yes 10:07:04 <strigazi> one sec 10:10:08 <strigazi> still looking 10:10:26 <strigazi> I don't remember were it is 10:11:33 <brtknr> i didnt know you could create master N G 10:11:35 <brtknr> i didnt know you could create master NG 10:11:57 <flwang1> yep, i thought master can only be in default-master NG 10:11:59 <strigazi> you can't atm, we have similar code though 10:12:00 <flwang1> that's why i asked 10:12:29 <flwang1> strigazi: can you please share the code with me when you found it? 10:12:35 <flwang1> i'm keen to learn that 10:13:35 <strigazi> found ir 10:13:36 <strigazi> found it 10:13:37 <strigazi> wait 10:14:19 <strigazi> https://github.com/openstack/magnum/blob/e52f77b299c50f004ee5a875c68d7f129b88a2af/magnum/conductor/handlers/cluster_conductor.py#L56 10:14:57 <strigazi> you can add the additional NGs there, needs some extra work for the LB to be shared across the stacks 10:15:26 <flwang1> hmm... what's the benefit to put masters into different NG? 10:15:46 <flwang1> anyway, thanks, i will think about this 10:15:47 <strigazi> to spawn across AZs? 10:15:57 <strigazi> that is the goal no? 10:16:26 <strigazi> Then you will have a NG with nodes in many AZs inside? 10:16:26 <flwang1> if there is an AZ list, we should be able to spawn masters into different AZ 10:16:50 <strigazi> So then all the nodegroup work we did is kind of worthless 10:17:12 <strigazi> as you want 10:17:24 <flwang1> strigazi: please don't say words like this 10:17:41 <flwang1> i'm not trying to make anyone's work worthless 10:18:02 <strigazi> The biggest use case of NGs was AZs 10:19:42 <flwang1> ok, then if you guys do have a good solution for this, please share, otherwise, what i am doing is trying to figure out a way to achieve that 10:20:15 <strigazi> well, I might be wrong. IN GKE they spawn AZs across zones, if I read correctly 10:20:18 <strigazi> https://cloud.google.com/kubernetes-engine/docs/concepts/node-pools 10:20:24 <strigazi> https://cloud.google.com/kubernetes-engine/docs/concepts/node-pools#nodes_in_multi-zonal_clusters 10:22:09 <strigazi> Maybe the thing with the list works 10:22:29 <flwang1> https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters 10:22:38 <strigazi> It will require some serious hacking in the heat templates, maybe 10:22:39 <flwang1> let's use a spec to track this work 10:22:53 <flwang1> i will start to draft that 10:23:24 <flwang1> brtknr: strigazi: could you please help review this one Volume AZ regression fix - https://review.opendev.org/705592 ? 10:23:47 <flwang1> it's kind of a regression issue, i didn't fix the volume AZ issue at the first shot 10:23:56 <strigazi> let's wrap the meeting? with train many bugs arrived 10:24:10 <flwang1> cinder doesn't respect "" as AZ 10:24:15 <flwang1> strigazi: sure 10:24:17 <strigazi> flwang1: I will have a look, I might have time only on monday though 10:24:23 <flwang1> #endmeeting