Wednesday, 2020-02-05

*** rcernin has joined #openstack-containers01:06
*** xinliang has joined #openstack-containers01:23
*** xinliang has quit IRC01:29
*** xinliang has joined #openstack-containers01:30
*** xinliang has quit IRC01:37
*** vesper has joined #openstack-containers01:38
*** vesper11 has quit IRC01:39
*** vishalmanchanda has joined #openstack-containers02:10
openstackgerritFeilong Wang proposed openstack/magnum master: [WIP][k8s]Upgrade calico/coredns to the latest stable version  https://review.opendev.org/70559903:10
openstackgerritFeilong Wang proposed openstack/magnum master: [WIP][k8s]Upgrade calico/coredns to the latest stable version  https://review.opendev.org/70559903:13
*** ykarel|away is now known as ykarel03:30
*** rcernin has quit IRC03:31
*** ramishra has joined #openstack-containers03:50
*** ramishra has quit IRC04:01
*** ramishra has joined #openstack-containers04:24
*** iokiwi has quit IRC04:45
*** udesale has joined #openstack-containers04:53
*** hongbin has joined #openstack-containers04:55
*** rcernin has joined #openstack-containers04:57
*** rcernin has quit IRC04:58
*** rcernin has joined #openstack-containers04:58
*** hongbin has quit IRC05:28
*** rcernin has quit IRC06:03
*** ykarel is now known as ykarel|lunch07:42
brtknrflwang: hi, just wondering if metric server works for you? Mine keeps doing a crashloopback07:56
*** elenalindq has joined #openstack-containers08:16
*** rcernin has joined #openstack-containers08:32
*** flwang1 has joined #openstack-containers08:35
*** rcernin has quit IRC08:48
*** ykarel|lunch is now known as ykarel08:52
*** rcernin has joined #openstack-containers08:54
*** xinliang has joined #openstack-containers08:58
strigazio/08:59
flwang1strigazi: hey08:59
strigazimeeting? no?09:00
strigazibrtknr yes it works09:01
flwang1#startmeeting magnum09:01
openstackMeeting started Wed Feb  5 09:01:39 2020 UTC and is due to finish in 60 minutes.  The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot.09:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.09:01
*** openstack changes topic to " (Meeting topic: magnum)"09:01
openstackThe meeting name has been set to 'magnum'09:01
flwang1#topic roll call09:01
*** openstack changes topic to "roll call (Meeting topic: magnum)"09:01
strigazio/09:01
flwang1o/09:01
flwang1brtknr: ?09:02
*** xinliang has quit IRC09:02
flwang1strigazi: so far i think just you and me09:02
flwang1strigazi: is there anything you want to discuss?09:03
brtknrflwang1: hi09:03
brtknro/09:03
brtknrwas scoffing down breakfast, apologies09:04
flwang1brtknr: all good09:04
flwang1brtknr: strigazi: do you guys have special topic to discuss?09:04
flwang1otherwise we go through the agenda?09:04
strigaziflwang1: brtknr: the new heat agent logs are worse09:04
brtknrelaborate09:05
strigaziflwang1: brtknr: the output of a command must be printed when it is executed09:05
brtknr+109:05
strigaziflwang1: brtknr: atm everything is printed at the end09:06
brtknrthere is some time where nothing gets printed09:06
brtknrwould be nice to fix that09:06
strigaziflwang1: brtknr: can't catch trhings that are stucj09:06
flwang1strigazi: yep, i notice that as well09:06
strigaziflwang1: brtknr: can't catch things that are stuck09:06
strigaziflwang1: brtknr: when did this happen?09:06
flwang1i'd like to see the logs being printed for each execution09:06
flwang1i think since last time brtknr fixed it?09:07
brtknrits been like this since we moved everything from cloud-init-output to journalctl09:07
brtknrwhen i fixed it, nothing was getting printed at all09:07
*** rcernin has quit IRC09:07
strigazi"nothing was getting printed at all" when was this true?09:07
brtknrwhen i fixed it, everything was getting printed on a single line09:07
strigaziok09:08
brtknri simply changed it so that newlines were parsed correctly09:08
strigaziI don't remember that. Anything we can do?09:08
flwang1nothing special, just review the code and see how can we fix it09:09
strigaziis it our code or os-collect-config?09:09
flwang1probably os-collect-config, i need to check the code, can't remember the details09:10
brtknrok who is going to look into this?09:13
flwang1i just created a story to track this https://storyboard.openstack.org/#!/story/200725609:14
flwang1strigazi: can you work on this?09:14
strigaziflwang1: I can have a look09:15
brtknrnext topic?09:16
strigaziwhere is the etherpad?09:17
flwang1https://etherpad.openstack.org/p/magnum-weekly-meeting09:17
strigaziit is not in he channel anymore09:17
strigazithx09:17
flwang1shall we go through the agenda?09:20
brtknryes09:20
brtknrare you guys happy to provide feedback for cinder_csi_enabled09:20
strigaziis it ready?09:21
brtknrto support out of tree cinder?09:21
brtknryep but i dont understand waht the pep8 failure is all about09:21
flwang1brtknr: so you have already tested and it works?09:21
flwang1i'm happy to test it09:21
brtknryep ive tested and it works09:22
flwang1great09:22
brtknrworks for xinliang too09:22
flwang1i will help take a look the pep8 and docs job failure09:23
strigazibrtknr: I will have a look, at CERN we have two more CSI drivers. Need to check that this is compatible.09:23
brtknrthere are some generic csi components, only one of them is cinder_csi replated09:25
brtknrrelated09:25
flwang1strigazi: if cern can give some comments on this csi patch, it would be great09:25
strigazibrtknr: don't you need any of: --feature-gates=CSINodeInfo=true,CSIDriverRegistry=true ?09:25
strigazibrtknr: or --runtime-config=storage.k8s.io/v1alpha1=true" ?09:25
brtknrdidnt need those09:26
brtknrseemed to work with v1.16.x and 1.17.x09:26
strigaziinteresting09:26
*** rcernin has joined #openstack-containers09:27
strigaziok, I will test09:27
brtknrCSIDriverRegistry went into beta from 1.1409:27
brtknrsame for CSINodeInfo09:27
brtknrso true by default09:27
strigazibrtknr: means on by default?09:27
strigaziok09:27
strigaziThe world will end based on a default09:28
brtknrXD09:28
brtknrI'll quote you on that when the world ends09:28
strigazi100% sure09:29
strigazisome will change the nuke_everything default to true in qa, and this it09:29
strigazianyway, let's stay on track, I will test it09:30
strigazibrtknr: would you like to do manila provisioner too?09:30
strigaziI'm good, let's move on?09:31
flwang1ok09:32
brtknrstrigazi: i tried the manila provisioner but lost my patience with it09:32
flwang1brtknr: what's the nginx issue?09:32
*** rcernin has quit IRC09:33
brtknrwith regards to nginx and traefik, at the moment, we need to label an ingress node. we wanted to explore adding another option which runs a deployment with n replicas behind a load balancer09:35
strigazibrtknr: what do you propose?09:36
brtknreffectively a label which changes nginx and traefik services from ClusterIP to LoadBalancer type09:37
brtknrfairly small change09:37
strigaziand DS to deploy and hostNetwork: false09:37
brtknrit can be ClusterIP by default09:37
strigaziand remove node-selector09:37
brtknrpossibly those too :S needs to look into it more closely09:39
strigazias mentioned before, just don't break the default :)09:40
brtknrok im glad you're okay to support the alternative, ill refine what it would involve a bit more in a PS09:41
strigazifor prometheus, ping dioguerra , I think it works but double check with him09:41
flwang1re prometheus, what's the status of removing heapster and adding metrics-server09:42
brtknri take back the promethus issue09:42
strigaziit is done09:42
flwang1i can see the metrics-server pods is broken on my local09:42
brtknrmetrics server doesnt work for me  either at 9.2.009:42
flwang1strigazi: you mean the heapster->metrics-server work is done, right09:43
flwang109:43
brtknrlemme grab the log09:43
strigaziyes09:43
strigazimetrics server works for me. Only the logs of the master node are not collected due to the cert09:43
flwang1strigazi: ok, i will test again09:44
flwang1what's the k8s version you're using?09:44
strigazi17.209:44
flwang1fairly new :)09:45
flwang1ok, i will give it a try09:45
flwang1let's move on?09:45
strigaziyes09:46
flwang1https://review.opendev.org/#/c/700565  work for ARM09:46
flwang1personally i'm ok with that09:46
brtknri created a story for the metrics-server issue: https://storyboard.openstack.org/#!/story/200726409:46
brtknrdo you need special label for metrics server?09:47
brtknrflwang1: have you tested  https://storyboard.openstack.org/#!/story/200726409:47
brtknrflwang1: have you tested https://review.opendev.org/#/c/70056509:47
strigazihttps://storyboard.openstack.org/#!/story/2007264 this is the heat-agent09:47
brtknri created a story for the metrics-server issue: https://storyboard.openstack.org/#!/story/200726509:48
brtknrsorry09:48
flwang1no i haven't, i just had a quick look about the code09:48
brtknrthe ARCH change looks trivial, i wanted to get your thoughts on the general approach09:49
strigazibrtknr: DNS k8s-podman-calico-k7tz6itxx7k6-node-0 on 10.254.0.10:53: no such host09:49
flwang1brtknr: will do09:49
strigazifor ARCH I will that the defailt x86_64 works09:49
brtknri.e. determining arch inside /write-heat-params.sh script09:50
strigaziI can't test arm09:50
brtknrstrigazi: what is the fix for DNS k8s-podman-calico-k7tz6itxx7k6-node-0 on 10.254.0.10:53: no such host09:51
strigazileft a comment09:51
brtknrstrigazi: ok thanks, that was not obvious to me :)09:53
strigazihttps://pbs.twimg.com/media/C9o9ZO0UwAAi3ib.jpg09:53
brtknrlol09:54
strigazidefaults usually assume DNS. I have more scenarios for the end of the world, all of them include DNS as well09:54
flwang1move on?09:55
flwang1we have only 5 mins09:55
jakeyiphi all09:56
flwang1brtknr: what's this 'Modify default-worker flavor after cluster creation'?09:56
flwang1jakeyip: hi09:56
brtknrflwang1: so we have a use case where we want to be able to change default worker flavor after the cluster is already creted09:58
brtknrat the moment, we can delete nodegroup and create a new nodegroup with different flavor09:58
brtknrbut we cant do this for default worker09:58
brtknri was wondering how involved a change it would be to allow update of default-worker flavor09:59
strigaziI think the best approach is to make the default worker deletable09:59
flwang1brtknr: maybe we should allow create an empty cluster10:00
flwang1like EKS10:00
strigazichanging the flavor sounds orthogonal to nova to me10:00
flwang1i don't think resizing the instance is a good idea10:00
strigazi+110:00
brtknryes i'd prefer to make the default worker deletable10:01
strigaziEKS logic gives little value10:01
brtknrand create cluster with 0 workers10:01
flwang1btw, i'd like to propose a patch to support distribute master nodes to different AZs, strigazi, do you like the idea?10:02
brtknrflwang1: is that what you mean by empty cluster ?10:02
flwang1brtknr: no, different idea10:02
brtknror does this also include 0 masters?10:02
jakeyipflwang1: +1 we will like that for workers10:02
flwang1it's like the Regional cluster in GKE10:02
flwang1jakeyip: for workers, we already can support it with node groups, but i know where you come from10:03
strigazimaster across AZs could work, but it feels hacky10:04
strigaziI mean you can deploy N NGs for masters by default10:04
strigazithis sounds better10:04
flwang1strigazi: what do you mean deploy N NGs for masters?10:05
strigaziif you have 3 AZs in your cloud, deploy three master NGs10:05
strigazion creation10:05
strigaziN=3 in this case10:06
brtknrthat sounds like a better approach10:06
flwang1you mean create 3NGs for master nodes?10:06
flwang1how can i do that?10:06
strigaziyes10:06
strigazione sec10:07
strigazistill looking10:10
strigaziI don't remember were it is10:10
brtknri didnt know you could create master N G10:11
brtknri didnt know you could create master NG10:11
flwang1yep, i thought master can only be in default-master NG10:11
strigaziyou can't atm, we have similar code though10:11
flwang1that's why i asked10:12
flwang1strigazi: can you please share the code with me when you found it?10:12
flwang1i'm keen to learn that10:12
strigazifound ir10:13
strigazifound it10:13
strigaziwait10:13
strigazihttps://github.com/openstack/magnum/blob/e52f77b299c50f004ee5a875c68d7f129b88a2af/magnum/conductor/handlers/cluster_conductor.py#L5610:14
strigaziyou can add the additional NGs there, needs some extra work for the LB to be shared across the stacks10:14
flwang1hmm... what's the benefit to put masters into different NG?10:15
*** ramishra has quit IRC10:15
flwang1anyway, thanks, i will think about this10:15
strigazito spawn across AZs?10:15
strigazithat is the goal no?10:15
strigaziThen you will have a NG with nodes in many AZs inside?10:16
flwang1if there is an AZ list, we should be able to spawn masters into different AZ10:16
strigaziSo then all the nodegroup work we did is kind of worthless10:16
strigazias you want10:17
flwang1strigazi: please don't say words like this10:17
flwang1i'm not trying to make anyone's work worthless10:17
strigaziThe biggest use case of NGs was AZs10:18
*** xinliang has joined #openstack-containers10:18
flwang1ok, then if you guys do have a good solution for this, please share, otherwise, what i am doing is trying to figure out a way to achieve that10:19
strigaziwell, I might be wrong. IN GKE they spawn AZs across zones, if I read correctly10:20
strigazihttps://cloud.google.com/kubernetes-engine/docs/concepts/node-pools10:20
strigazihttps://cloud.google.com/kubernetes-engine/docs/concepts/node-pools#nodes_in_multi-zonal_clusters10:20
*** ramishra has joined #openstack-containers10:21
strigaziMaybe the thing with the list works10:22
flwang1https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters10:22
strigaziIt will require some serious hacking in the heat templates, maybe10:22
flwang1let's use a spec to track this work10:22
flwang1i will start to draft that10:22
flwang1brtknr: strigazi: could you please help review this one   Volume AZ regression fix - https://review.opendev.org/705592 ?10:23
flwang1it's kind of a regression issue, i didn't fix the volume AZ issue at the first shot10:23
strigazilet's wrap the meeting? with train many bugs arrived10:23
flwang1cinder doesn't respect "" as AZ10:24
flwang1strigazi: sure10:24
strigaziflwang1: I will have a look, I might have time only on monday though10:24
flwang1#endmeeting10:24
*** openstack changes topic to "OpenStack Containers Team | Meeting: every Wednesday @ 9AM UTC | Agenda: https://etherpad.openstack.org/p/magnum-weekly-meeting"10:24
openstackMeeting ended Wed Feb  5 10:24:23 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)10:24
openstackMinutes:        http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-02-05-09.01.html10:24
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-02-05-09.01.txt10:24
openstackLog:            http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-02-05-09.01.log.html10:24
flwang1strigazi: thanks, it would be really appreciated10:24
flwang1share on me, i didn't get it fully tested10:24
jakeyipthe volume az patch is similar to one of our custom patch10:24
flwang1jakeyip: thanks, if you can, please help review it10:25
jakeyipnot sure if it's totally correct though... that's why I didn't submit it upstream10:25
jakeyipcinder AZ may not be equal to nova AZ10:25
flwang1jakeyip: we have to use the condition, since cinder is not smart enough10:25
* strigazi goes bug hunting10:26
flwang1jakeyip: yep, i know, it's a good point.10:26
jakeyipI'm referring more to the original patch rather than the regression. I agree with the regression fix. The logic for the original patch may not be correct globally10:26
flwang1jakeyip: but most of the companies will design it to make sure they're same to avoid troubles10:26
jakeyipIt is OK for us so we carry such a patch in house ( -volume-type +availability-zone)10:27
jakeyipit might not be OK for the world10:27
jakeyipwhat is needed might be a way to override Cinder AZ10:27
flwang1jakeyip: it would be a bit overkill if we introduce another label for cinder az :(10:27
jakeyiptake it from the AZ parameter but allow override if possible10:28
jakeyipyeap. hence also the note I put on agenda10:28
flwang1i'm going to introduce an AZ list as config option10:28
jakeyipjust suggestions don't have to implement until someone files a bug report :P10:28
flwang1if it's set, magnum will randomly pick one from the list10:28
jakeyipour patch - https://github.com/NeCTAR-RC/magnum/commit/e3e4afe10bf22aee7824dfc0df5d30b2c55502e810:29
flwang1jakeyip: labels are an old topic and we don't have a perfect answer yet10:29
flwang1jakeyip: that's a lazy patch :)10:30
jakeyipthat's why it's not upstream :P10:30
flwang1jakeyip: i have to go10:30
jakeyipseeya10:30
flwang1jakeyip: please help review the patch, we need you guys input10:31
jakeyipok10:31
flwang1thanks10:32
flwang1ttyl10:32
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: More verbose logs for cluster ops  https://review.opendev.org/70592210:50
*** udesale_ has joined #openstack-containers11:13
*** udesale has quit IRC11:16
*** ramishra has quit IRC11:18
*** ramishra has joined #openstack-containers11:22
brtknrstrigazi: im trying to work out how i can create a cluster with 0 workers11:23
*** pcaruana has quit IRC11:37
*** pcaruana has joined #openstack-containers11:50
openstackgerritBharat Kunwar proposed openstack/magnum master: Add cinder_csi_enabled label  https://review.opendev.org/70035712:05
*** lpetrut has joined #openstack-containers13:24
*** ykarel is now known as ykarel|away13:27
*** pcaruana has quit IRC14:17
*** ykarel|away is now known as ykarel14:20
*** ykarel is now known as ykarel|away15:08
*** jmlowe has joined #openstack-containers15:13
*** jmlowe has quit IRC15:17
*** jmlowe has joined #openstack-containers15:17
openstackgerritBharat Kunwar proposed openstack/magnum master: [k8s] Fix metrics-server CrashLoopBack  https://review.opendev.org/70598415:25
*** lpetrut has quit IRC15:28
*** ramishra has quit IRC15:37
*** ianychoi_ is now known as ianychoi15:40
*** osmanlicilegi has quit IRC15:42
*** osmanlicilegi has joined #openstack-containers15:42
*** pcaruana has joined #openstack-containers15:49
*** openstackstatus has joined #openstack-containers16:04
*** ChanServ sets mode: +v openstackstatus16:04
*** jmlowe has quit IRC16:08
*** ramishra has joined #openstack-containers16:16
*** kklimonda has quit IRC16:22
*** kklimonda has joined #openstack-containers16:22
*** udesale_ has quit IRC16:24
*** jmlowe has joined #openstack-containers16:47
*** flwang has quit IRC17:48
*** NobodyCam has quit IRC17:50
*** NobodyCam has joined #openstack-containers17:51
*** jmlowe has quit IRC17:54
*** NobodyCam has quit IRC19:39
*** kklimonda has quit IRC19:40
*** jmlowe has joined #openstack-containers20:00
*** jmlowe has quit IRC20:01
*** jmlowe has joined #openstack-containers20:15
*** mgariepy has quit IRC20:54
*** jmlowe has quit IRC21:08
*** jmlowe has joined #openstack-containers21:10
*** jmlowe has quit IRC21:22
*** elenalindq has quit IRC21:30
*** yolanda has joined #openstack-containers22:35
*** kklimonda has joined #openstack-containers22:59
*** NobodyCam has joined #openstack-containers23:00

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!